feat: add intelligent import for CSV, Excel, and PDF files
Add IntelligentImportAugmentation with support for: - CSV: auto-detection of encoding, delimiters, and field types - Excel: multi-sheet extraction with metadata preservation - PDF: text extraction, table detection, and metadata extraction Features: - Automatic format detection from file extension or content - Intelligent type inference (string, number, boolean, date) - Seamless integration with neural entity extraction - Production-ready with 69 comprehensive tests Dependencies added: - xlsx@^0.18.5 for Excel parsing - pdfjs-dist@^4.0.379 for PDF parsing - csv-parse@^6.1.0 for CSV parsing - chardet@^2.0.0 for encoding detection Documentation: - Updated README with import examples - Updated API_REFERENCE with comprehensive import() docs - Updated import-anything.md guide - Added working example: import-excel-pdf-csv.ts - Updated augmentations README
This commit is contained in:
parent
aaf8e0f411
commit
814cbb48ee
33 changed files with 4664 additions and 28 deletions
40
README.md
40
README.md
|
|
@ -457,6 +457,41 @@ npm run download-models # Download Q8 model
|
|||
npm run download-models:q8 # Download Q8 model
|
||||
```
|
||||
|
||||
## 🚀 Import Anything - Files, Data, URLs
|
||||
|
||||
Brainy's universal import intelligently handles **any data format**:
|
||||
|
||||
```javascript
|
||||
// Import CSV with auto-detection
|
||||
await brain.import('customers.csv')
|
||||
// ✨ Auto-detects: encoding, delimiter, types, creates entities!
|
||||
|
||||
// Import Excel workbooks with multi-sheet support
|
||||
await brain.import('sales-data.xlsx', {
|
||||
excelSheets: ['Q1', 'Q2'] // or 'all' for all sheets
|
||||
})
|
||||
// ✨ Processes all sheets, preserves structure, infers types!
|
||||
|
||||
// Import PDF documents with table extraction
|
||||
await brain.import('research-paper.pdf', {
|
||||
pdfExtractTables: true
|
||||
})
|
||||
// ✨ Extracts text, detects tables, preserves metadata!
|
||||
|
||||
// Import JSON/YAML data
|
||||
await brain.import([
|
||||
{ name: 'Alice', role: 'Engineer' },
|
||||
{ name: 'Bob', role: 'Designer' }
|
||||
])
|
||||
// ✨ Automatically creates Person entities with relationships!
|
||||
|
||||
// Import from URLs (auto-fetched)
|
||||
await brain.import('https://api.example.com/data.json')
|
||||
// ✨ Auto-detects URL, fetches, parses, processes!
|
||||
```
|
||||
|
||||
**📖 [Complete Import Guide →](docs/guides/import-anything.md)** | **[Live Example →](examples/import-excel-pdf-csv.ts)**
|
||||
|
||||
## 📚 Core API
|
||||
|
||||
### `search()` - Vector Similarity
|
||||
|
|
@ -513,6 +548,11 @@ await brain.deleteVerb(verbId)
|
|||
// Bulk operations
|
||||
await brain.import(arrayOfData)
|
||||
const exported = await brain.export({format: 'json'})
|
||||
|
||||
// Import from CSV, Excel, PDF files (auto-detected)
|
||||
await brain.import('customers.csv') // CSV with encoding detection
|
||||
await brain.import('sales-report.xlsx') // Excel with multi-sheet support
|
||||
await brain.import('research.pdf') // PDF with table extraction
|
||||
```
|
||||
|
||||
## 🌐 Distributed System (NEW!)
|
||||
|
|
|
|||
|
|
@ -683,26 +683,75 @@ await data.clear({
|
|||
---
|
||||
|
||||
### `async import(params): Promise<ImportResult>`
|
||||
Imports data from various formats.
|
||||
Imports data from various formats with automatic detection.
|
||||
|
||||
**Parameters:**
|
||||
- `data` (required) - Data to import
|
||||
- `format` (required) - 'json' | 'csv' | 'parquet'
|
||||
- `mapping` - Field mapping
|
||||
- `batchSize` - Import batch size
|
||||
- `validate` - Validate items
|
||||
- `data` (required) - Data to import (Buffer, string, array, or file path)
|
||||
- `format` - 'auto' | 'json' | 'csv' | 'excel' | 'pdf' | 'yaml' | 'text' (default: 'auto')
|
||||
- `mapping` - Field mapping configuration
|
||||
- `batchSize` - Import batch size (default: 50)
|
||||
- `validate` - Validate items before import
|
||||
- `relationships` - Extract relationships automatically (default: true)
|
||||
|
||||
**Example:**
|
||||
**CSV-specific options:**
|
||||
- `csvDelimiter` - Column delimiter (auto-detected if not specified)
|
||||
- `csvHeaders` - First row contains headers (default: true)
|
||||
- `encoding` - Character encoding (auto-detected if not specified)
|
||||
|
||||
**Excel-specific options:**
|
||||
- `excelSheets` - Sheet names array or 'all' for all sheets
|
||||
|
||||
**PDF-specific options:**
|
||||
- `pdfExtractTables` - Extract tables from PDFs (default: true)
|
||||
- `pdfPreserveLayout` - Preserve text layout (default: true)
|
||||
|
||||
**Examples:**
|
||||
```typescript
|
||||
const result = await data.import({
|
||||
data: csvData,
|
||||
format: 'csv',
|
||||
mapping: {
|
||||
'name': 'title',
|
||||
'desc': 'description'
|
||||
},
|
||||
batchSize: 500
|
||||
// Import CSV file with auto-detection
|
||||
const csvResult = await brain.import('customers.csv')
|
||||
// Auto-detects: format, encoding, delimiter, field types
|
||||
|
||||
// Import Excel workbook
|
||||
const excelResult = await brain.import('sales-data.xlsx', {
|
||||
excelSheets: ['Q1', 'Q2'] // Import specific sheets
|
||||
})
|
||||
|
||||
// Import PDF with table extraction
|
||||
const pdfResult = await brain.import('report.pdf', {
|
||||
pdfExtractTables: true
|
||||
})
|
||||
|
||||
// Import data array
|
||||
const dataResult = await brain.import([
|
||||
{ name: 'Alice', role: 'Engineer' },
|
||||
{ name: 'Bob', role: 'Designer' }
|
||||
], {
|
||||
batchSize: 100,
|
||||
relationships: true // Auto-extract relationships
|
||||
})
|
||||
|
||||
// Import with custom CSV delimiter
|
||||
const tsvResult = await brain.import('data.tsv', {
|
||||
format: 'csv',
|
||||
csvDelimiter: '\t'
|
||||
})
|
||||
```
|
||||
|
||||
**Returns:**
|
||||
```typescript
|
||||
{
|
||||
success: boolean
|
||||
imported: number // Number of items successfully imported
|
||||
failed: number // Number of items that failed
|
||||
entityIds: string[] // IDs of created entities
|
||||
metadata: {
|
||||
format: string // Detected format
|
||||
encoding?: string // Detected encoding (CSV)
|
||||
delimiter?: string // Detected delimiter (CSV)
|
||||
sheets?: string[] // Processed sheets (Excel)
|
||||
pageCount?: number // Number of pages (PDF)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ Brainy's import is **ONE magical method** that understands EVERYTHING:
|
|||
- 📊 Data (objects, arrays, strings)
|
||||
- 📁 Files (auto-detects by path)
|
||||
- 🌐 URLs (auto-fetches)
|
||||
- 📄 Formats (JSON, CSV, YAML, text - all auto-detected)
|
||||
- 📄 Formats (JSON, CSV, Excel, PDF, YAML, text - all auto-detected)
|
||||
|
||||
## The Ultimate Simplicity
|
||||
|
||||
|
|
@ -36,7 +36,7 @@ await brain.import(people)
|
|||
```javascript
|
||||
// From file? Just pass the path!
|
||||
await brain.import('customers.csv')
|
||||
// ✨ Auto-detects it's a file, parses CSV, creates entities!
|
||||
// ✨ Auto-detects encoding, delimiter, types - creates entities!
|
||||
|
||||
// Or pass CSV content directly
|
||||
const csv = `name,age,city
|
||||
|
|
@ -47,6 +47,32 @@ await brain.import(csv, { format: 'csv' })
|
|||
// ✨ Smart CSV parsing handles quotes, escapes, everything!
|
||||
```
|
||||
|
||||
### 📊 Import Excel - Multi-Sheet Support
|
||||
```javascript
|
||||
// Import entire Excel workbook
|
||||
await brain.import('sales-report.xlsx')
|
||||
// ✨ Processes all sheets, preserves structure, infers types!
|
||||
|
||||
// Or specific sheets only
|
||||
await brain.import('data.xlsx', {
|
||||
excelSheets: ['Customers', 'Orders']
|
||||
})
|
||||
// ✨ Multi-sheet data becomes interconnected entities!
|
||||
```
|
||||
|
||||
### 📑 Import PDF - Text & Tables
|
||||
```javascript
|
||||
// Import PDF documents
|
||||
await brain.import('research-paper.pdf')
|
||||
// ✨ Extracts text, detects tables, preserves metadata!
|
||||
|
||||
// With table extraction
|
||||
await brain.import('report.pdf', {
|
||||
pdfExtractTables: true
|
||||
})
|
||||
// ✨ Converts PDF tables to structured data automatically!
|
||||
```
|
||||
|
||||
### 📝 Import YAML - File or String
|
||||
```javascript
|
||||
// From file? Auto-detected!
|
||||
|
|
@ -92,11 +118,12 @@ await brain.import(article, { format: 'text' })
|
|||
|
||||
When you import data, Brainy:
|
||||
|
||||
1. **Auto-detects format** - JSON, CSV, YAML, text, or by file extension
|
||||
2. **Identifies entity types** - Uses AI to classify as Person, Document, Product, etc. (31 types!)
|
||||
3. **Finds relationships** - Detects connections like "belongsTo", "createdBy", "references" (40 types!)
|
||||
4. **Creates embeddings** - Makes everything semantically searchable
|
||||
5. **Indexes metadata** - Enables lightning-fast filtering
|
||||
1. **Auto-detects format** - CSV, Excel, PDF, JSON, YAML, text, or by file extension
|
||||
2. **Intelligent parsing** - CSV (encoding/delimiter detection), Excel (multi-sheet), PDF (text/tables)
|
||||
3. **Identifies entity types** - Uses AI to classify as Person, Document, Product, etc. (31 types!)
|
||||
4. **Finds relationships** - Detects connections like "belongsTo", "createdBy", "references" (40 types!)
|
||||
5. **Creates embeddings** - Makes everything semantically searchable
|
||||
6. **Indexes metadata** - Enables lightning-fast filtering
|
||||
|
||||
## Intelligent Type Detection
|
||||
|
||||
|
|
@ -160,9 +187,21 @@ Everything works with zero config, but you can customize:
|
|||
|
||||
```javascript
|
||||
await brain.import(data, {
|
||||
format: 'auto', // 'json' | 'csv' | 'yaml' | 'text' | 'auto'
|
||||
batchSize: 50, // Process in batches
|
||||
relationships: true // Extract relationships (default: true)
|
||||
format: 'auto', // 'csv' | 'excel' | 'pdf' | 'json' | 'yaml' | 'text' | 'auto'
|
||||
batchSize: 50, // Process in batches
|
||||
relationships: true, // Extract relationships (default: true)
|
||||
|
||||
// CSV-specific options
|
||||
csvDelimiter: ',', // Auto-detected if not specified
|
||||
csvHeaders: true, // First row is headers
|
||||
encoding: 'utf-8', // Auto-detected if not specified
|
||||
|
||||
// Excel-specific options
|
||||
excelSheets: ['Sheet1'], // or 'all' for all sheets
|
||||
|
||||
// PDF-specific options
|
||||
pdfExtractTables: true, // Extract tables from PDFs
|
||||
pdfPreserveLayout: true // Preserve text layout
|
||||
})
|
||||
```
|
||||
|
||||
|
|
|
|||
148
examples/import-excel-pdf-csv.ts
Normal file
148
examples/import-excel-pdf-csv.ts
Normal file
|
|
@ -0,0 +1,148 @@
|
|||
/**
|
||||
* Intelligent Import Example
|
||||
*
|
||||
* Demonstrates importing CSV, Excel, and PDF files with automatic:
|
||||
* - Format detection
|
||||
* - Type inference
|
||||
* - Entity extraction
|
||||
* - Relationship detection
|
||||
*/
|
||||
|
||||
import { Brainy } from '../src/brainy.js'
|
||||
import { promises as fs } from 'fs'
|
||||
import * as path from 'path'
|
||||
|
||||
async function main() {
|
||||
console.log('🧠 Brainy Intelligent Import Example\n')
|
||||
|
||||
const brain = new Brainy({ verbose: false })
|
||||
await brain.init()
|
||||
|
||||
console.log('✅ Brainy initialized\n')
|
||||
|
||||
// Example 1: Import CSV file
|
||||
console.log('📄 Example 1: Import CSV File')
|
||||
console.log('─────────────────────────────\n')
|
||||
|
||||
const csvPath = path.join(process.cwd(), 'tests/fixtures/import/simple.csv')
|
||||
const csvExists = await fs.access(csvPath).then(() => true).catch(() => false)
|
||||
|
||||
if (csvExists) {
|
||||
const csvBuffer = await fs.readFile(csvPath)
|
||||
|
||||
const csvResult = await brain.import(csvBuffer, {
|
||||
filename: 'simple.csv',
|
||||
format: 'auto' // Auto-detects as CSV
|
||||
})
|
||||
|
||||
console.log(`✨ Imported CSV with ${csvResult.length || 'structured'} data`)
|
||||
console.log(' Auto-detected: delimiter, encoding, types')
|
||||
console.log(' Extracted: entities with proper typing\n')
|
||||
}
|
||||
|
||||
// Example 2: Import Excel file
|
||||
console.log('📊 Example 2: Import Excel Workbook')
|
||||
console.log('────────────────────────────────────\n')
|
||||
|
||||
const excelPath = path.join(process.cwd(), 'tests/fixtures/import/multi-sheet.xlsx')
|
||||
const excelExists = await fs.access(excelPath).then(() => true).catch(() => false)
|
||||
|
||||
if (excelExists) {
|
||||
const excelBuffer = await fs.readFile(excelPath)
|
||||
|
||||
const excelResult = await brain.import(excelBuffer, {
|
||||
filename: 'multi-sheet.xlsx',
|
||||
format: 'auto', // Auto-detects as Excel
|
||||
excelSheets: 'all' // Process all sheets
|
||||
})
|
||||
|
||||
console.log(`✨ Imported Excel workbook`)
|
||||
console.log(' Processed: Multiple sheets')
|
||||
console.log(' Extracted: Structured data from each sheet')
|
||||
console.log(' Preserved: Sheet metadata and relationships\n')
|
||||
}
|
||||
|
||||
// Example 3: Import PDF file
|
||||
console.log('📑 Example 3: Import PDF Document')
|
||||
console.log('─────────────────────────────────\n')
|
||||
|
||||
const pdfPath = path.join(process.cwd(), 'tests/fixtures/import/simple.pdf')
|
||||
const pdfExists = await fs.access(pdfPath).then(() => true).catch(() => false)
|
||||
|
||||
if (pdfExists) {
|
||||
const pdfBuffer = await fs.readFile(pdfPath)
|
||||
|
||||
const pdfResult = await brain.import(pdfBuffer, {
|
||||
filename: 'simple.pdf',
|
||||
format: 'auto', // Auto-detects as PDF
|
||||
pdfExtractTables: true
|
||||
})
|
||||
|
||||
console.log(`✨ Imported PDF document`)
|
||||
console.log(' Extracted: Text content with layout preservation')
|
||||
console.log(' Detected: Tables (if present)')
|
||||
console.log(' Preserved: Metadata (author, title, dates)\n')
|
||||
}
|
||||
|
||||
// Example 4: Import with specific sheet selection
|
||||
console.log('🎯 Example 4: Selective Sheet Import')
|
||||
console.log('────────────────────────────────────\n')
|
||||
|
||||
if (excelExists) {
|
||||
const excelBuffer = await fs.readFile(excelPath)
|
||||
|
||||
await brain.import(excelBuffer, {
|
||||
filename: 'multi-sheet.xlsx',
|
||||
excelSheets: ['Products'] // Only import Products sheet
|
||||
})
|
||||
|
||||
console.log(`✨ Imported specific Excel sheet`)
|
||||
console.log(' Sheet: Products only')
|
||||
console.log(' Benefit: Faster processing, focused data\n')
|
||||
}
|
||||
|
||||
// Example 5: CSV with custom delimiter
|
||||
console.log('⚙️ Example 5: CSV with Custom Delimiter')
|
||||
console.log('───────────────────────────────────────\n')
|
||||
|
||||
const tsvPath = path.join(process.cwd(), 'tests/fixtures/import/tab-delimited.csv')
|
||||
const tsvExists = await fs.access(tsvPath).then(() => true).catch(() => false)
|
||||
|
||||
if (tsvExists) {
|
||||
const tsvBuffer = await fs.readFile(tsvPath)
|
||||
|
||||
await brain.import(tsvBuffer, {
|
||||
filename: 'tab-delimited.csv',
|
||||
format: 'csv',
|
||||
csvDelimiter: '\t' // Or let it auto-detect
|
||||
})
|
||||
|
||||
console.log(`✨ Imported tab-delimited file`)
|
||||
console.log(' Delimiter: Auto-detected (tab)')
|
||||
console.log(' Works with: comma, semicolon, tab, pipe\n')
|
||||
}
|
||||
|
||||
// Example 6: Query imported data
|
||||
console.log('🔍 Example 6: Query Imported Data')
|
||||
console.log('──────────────────────────────────\n')
|
||||
|
||||
const results = await brain.search('product', { limit: 5 })
|
||||
console.log(`Found ${results.length} results for "product"`)
|
||||
|
||||
if (results.length > 0) {
|
||||
console.log('Sample result:')
|
||||
const sample = results[0]
|
||||
console.log(` ID: ${sample.id}`)
|
||||
console.log(` Data: ${JSON.stringify(sample.data).slice(0, 100)}...`)
|
||||
}
|
||||
|
||||
console.log('\n✨ Example Complete!')
|
||||
console.log('\n📚 Key Takeaways:')
|
||||
console.log(' • One method (brain.import) handles CSV, Excel, and PDF')
|
||||
console.log(' • Format auto-detection from file extension or content')
|
||||
console.log(' • Intelligent parsing: encoding, delimiters, types, tables')
|
||||
console.log(' • Zero config required - works out of the box')
|
||||
console.log(' • All data becomes searchable via Triple Intelligence')
|
||||
}
|
||||
|
||||
main().catch(console.error)
|
||||
550
package-lock.json
generated
550
package-lock.json
generated
|
|
@ -13,13 +13,17 @@
|
|||
"@huggingface/transformers": "^3.7.2",
|
||||
"boxen": "^8.0.1",
|
||||
"chalk": "^5.3.0",
|
||||
"chardet": "^2.0.0",
|
||||
"cli-table3": "^0.6.5",
|
||||
"commander": "^11.1.0",
|
||||
"csv-parse": "^6.1.0",
|
||||
"inquirer": "^12.9.3",
|
||||
"ora": "^8.2.0",
|
||||
"pdfjs-dist": "^4.0.379",
|
||||
"prompts": "^2.4.2",
|
||||
"uuid": "^9.0.1",
|
||||
"ws": "^8.18.3"
|
||||
"ws": "^8.18.3",
|
||||
"xlsx": "^0.18.5"
|
||||
},
|
||||
"bin": {
|
||||
"brainy": "bin/brainy.js"
|
||||
|
|
@ -36,6 +40,7 @@
|
|||
"@typescript-eslint/eslint-plugin": "^8.0.0",
|
||||
"@typescript-eslint/parser": "^8.0.0",
|
||||
"@vitest/coverage-v8": "^3.2.4",
|
||||
"jspdf": "^3.0.3",
|
||||
"minio": "^8.0.5",
|
||||
"standard-version": "^9.5.0",
|
||||
"testcontainers": "^11.5.1",
|
||||
|
|
@ -983,6 +988,16 @@
|
|||
"node": ">=6.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/runtime": {
|
||||
"version": "7.28.4",
|
||||
"resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.28.4.tgz",
|
||||
"integrity": "sha512-Q/N6JNWvIvPnLDvjlE1OUBLPQHH6l3CltCEsHIujp45zQUSSh8K+gHnaEX45yAT1nyngnINhvWtzN+Nb9D8RAQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=6.9.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/types": {
|
||||
"version": "7.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@babel/types/-/types-7.28.2.tgz",
|
||||
|
|
@ -2846,6 +2861,191 @@
|
|||
"url": "https://opencollective.com/js-sdsl"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/canvas": {
|
||||
"version": "0.1.80",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/canvas/-/canvas-0.1.80.tgz",
|
||||
"integrity": "sha512-DxuT1ClnIPts1kQx8FBmkk4BQDTfI5kIzywAaMjQSXfNnra5UFU9PwurXrl+Je3bJ6BGsp/zmshVVFbCmyI+ww==",
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"workspaces": [
|
||||
"e2e/*"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@napi-rs/canvas-android-arm64": "0.1.80",
|
||||
"@napi-rs/canvas-darwin-arm64": "0.1.80",
|
||||
"@napi-rs/canvas-darwin-x64": "0.1.80",
|
||||
"@napi-rs/canvas-linux-arm-gnueabihf": "0.1.80",
|
||||
"@napi-rs/canvas-linux-arm64-gnu": "0.1.80",
|
||||
"@napi-rs/canvas-linux-arm64-musl": "0.1.80",
|
||||
"@napi-rs/canvas-linux-riscv64-gnu": "0.1.80",
|
||||
"@napi-rs/canvas-linux-x64-gnu": "0.1.80",
|
||||
"@napi-rs/canvas-linux-x64-musl": "0.1.80",
|
||||
"@napi-rs/canvas-win32-x64-msvc": "0.1.80"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/canvas-android-arm64": {
|
||||
"version": "0.1.80",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/canvas-android-arm64/-/canvas-android-arm64-0.1.80.tgz",
|
||||
"integrity": "sha512-sk7xhN/MoXeuExlggf91pNziBxLPVUqF2CAVnB57KLG/pz7+U5TKG8eXdc3pm0d7Od0WreB6ZKLj37sX9muGOQ==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"android"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/canvas-darwin-arm64": {
|
||||
"version": "0.1.80",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/canvas-darwin-arm64/-/canvas-darwin-arm64-0.1.80.tgz",
|
||||
"integrity": "sha512-O64APRTXRUiAz0P8gErkfEr3lipLJgM6pjATwavZ22ebhjYl/SUbpgM0xcWPQBNMP1n29afAC/Us5PX1vg+JNQ==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/canvas-darwin-x64": {
|
||||
"version": "0.1.80",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/canvas-darwin-x64/-/canvas-darwin-x64-0.1.80.tgz",
|
||||
"integrity": "sha512-FqqSU7qFce0Cp3pwnTjVkKjjOtxMqRe6lmINxpIZYaZNnVI0H5FtsaraZJ36SiTHNjZlUB69/HhxNDT1Aaa9vA==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/canvas-linux-arm-gnueabihf": {
|
||||
"version": "0.1.80",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-arm-gnueabihf/-/canvas-linux-arm-gnueabihf-0.1.80.tgz",
|
||||
"integrity": "sha512-eyWz0ddBDQc7/JbAtY4OtZ5SpK8tR4JsCYEZjCE3dI8pqoWUC8oMwYSBGCYfsx2w47cQgQCgMVRVTFiiO38hHQ==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/canvas-linux-arm64-gnu": {
|
||||
"version": "0.1.80",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-arm64-gnu/-/canvas-linux-arm64-gnu-0.1.80.tgz",
|
||||
"integrity": "sha512-qwA63t8A86bnxhuA/GwOkK3jvb+XTQaTiVML0vAWoHyoZYTjNs7BzoOONDgTnNtr8/yHrq64XXzUoLqDzU+Uuw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/canvas-linux-arm64-musl": {
|
||||
"version": "0.1.80",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-arm64-musl/-/canvas-linux-arm64-musl-0.1.80.tgz",
|
||||
"integrity": "sha512-1XbCOz/ymhj24lFaIXtWnwv/6eFHXDrjP0jYkc6iHQ9q8oXKzUX1Lc6bu+wuGiLhGh2GS/2JlfORC5ZcXimRcg==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/canvas-linux-riscv64-gnu": {
|
||||
"version": "0.1.80",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-riscv64-gnu/-/canvas-linux-riscv64-gnu-0.1.80.tgz",
|
||||
"integrity": "sha512-XTzR125w5ZMs0lJcxRlS1K3P5RaZ9RmUsPtd1uGt+EfDyYMu4c6SEROYsxyatbbu/2+lPe7MPHOO/0a0x7L/gw==",
|
||||
"cpu": [
|
||||
"riscv64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/canvas-linux-x64-gnu": {
|
||||
"version": "0.1.80",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-x64-gnu/-/canvas-linux-x64-gnu-0.1.80.tgz",
|
||||
"integrity": "sha512-BeXAmhKg1kX3UCrJsYbdQd3hIMDH/K6HnP/pG2LuITaXhXBiNdh//TVVVVCBbJzVQaV5gK/4ZOCMrQW9mvuTqA==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/canvas-linux-x64-musl": {
|
||||
"version": "0.1.80",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-x64-musl/-/canvas-linux-x64-musl-0.1.80.tgz",
|
||||
"integrity": "sha512-x0XvZWdHbkgdgucJsRxprX/4o4sEed7qo9rCQA9ugiS9qE2QvP0RIiEugtZhfLH3cyI+jIRFJHV4Fuz+1BHHMg==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/canvas-win32-x64-msvc": {
|
||||
"version": "0.1.80",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/canvas-win32-x64-msvc/-/canvas-win32-x64-msvc-0.1.80.tgz",
|
||||
"integrity": "sha512-Z8jPsM6df5V8B1HrCHB05+bDiCxjE9QA//3YrkKIdVDEwn5RKaqOxCJDRJkl48cJbylcrJbW4HxZbTte8juuPg==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">= 10"
|
||||
}
|
||||
},
|
||||
"node_modules/@nodelib/fs.scandir": {
|
||||
"version": "2.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@nodelib/fs.scandir/-/fs.scandir-2.1.5.tgz",
|
||||
|
|
@ -4181,6 +4381,21 @@
|
|||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/pako": {
|
||||
"version": "2.0.4",
|
||||
"resolved": "https://registry.npmjs.org/@types/pako/-/pako-2.0.4.tgz",
|
||||
"integrity": "sha512-VWDCbrLeVXJM9fihYodcLiIv0ku+AlOa/TQ1SvYOaBuyrSKgEcro95LJyIsJ4vSo6BXIxOKxiJAat04CmST9Fw==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/raf": {
|
||||
"version": "3.4.3",
|
||||
"resolved": "https://registry.npmjs.org/@types/raf/-/raf-3.4.3.tgz",
|
||||
"integrity": "sha512-c4YAvMedbPZ5tEyxzQdMoOhhJ4RD3rngZIdwC2/qDN3d7JpEhB6fiBRKVY1lg5B7Wk+uPBjn5f39j1/2MY1oOw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true
|
||||
},
|
||||
"node_modules/@types/resolve": {
|
||||
"version": "1.20.2",
|
||||
"resolved": "https://registry.npmjs.org/@types/resolve/-/resolve-1.20.2.tgz",
|
||||
|
|
@ -4225,6 +4440,14 @@
|
|||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/trusted-types": {
|
||||
"version": "2.0.7",
|
||||
"resolved": "https://registry.npmjs.org/@types/trusted-types/-/trusted-types-2.0.7.tgz",
|
||||
"integrity": "sha512-ScaPdn1dQczgbl0QFTeTOmVHFULt394XJgOQNoyVhZ6r2vLnMLJfBPd53SB52T/3G36VI1/g2MZaX0cwDuXsfw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true
|
||||
},
|
||||
"node_modules/@types/uuid": {
|
||||
"version": "10.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@types/uuid/-/uuid-10.0.0.tgz",
|
||||
|
|
@ -4688,6 +4911,15 @@
|
|||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/adler-32": {
|
||||
"version": "1.3.1",
|
||||
"resolved": "https://registry.npmjs.org/adler-32/-/adler-32-1.3.1.tgz",
|
||||
"integrity": "sha512-ynZ4w/nUUv5rrsR8UUGoe1VC9hZj6V5hU9Qw1HlMDJGEJw5S7TfTErWTjMys6M7vr0YWcPqs3qAr4ss0nDfP+A==",
|
||||
"license": "Apache-2.0",
|
||||
"engines": {
|
||||
"node": ">=0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/ajv": {
|
||||
"version": "6.12.6",
|
||||
"resolved": "https://registry.npmjs.org/ajv/-/ajv-6.12.6.tgz",
|
||||
|
|
@ -5083,6 +5315,17 @@
|
|||
"bare-path": "^3.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/base64-arraybuffer": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/base64-arraybuffer/-/base64-arraybuffer-1.0.2.tgz",
|
||||
"integrity": "sha512-I3yl4r9QB5ZRY3XuJVEPfc2XhZO6YweFPI+UovAzn+8/hb3oJ6lnysaFcjVpkCPfVWFUDvoZ8kmVDP7WyRtYtQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"engines": {
|
||||
"node": ">= 0.6.0"
|
||||
}
|
||||
},
|
||||
"node_modules/base64-js": {
|
||||
"version": "1.5.1",
|
||||
"resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
|
||||
|
|
@ -5451,6 +5694,40 @@
|
|||
"node": ">=6"
|
||||
}
|
||||
},
|
||||
"node_modules/canvg": {
|
||||
"version": "3.0.11",
|
||||
"resolved": "https://registry.npmjs.org/canvg/-/canvg-3.0.11.tgz",
|
||||
"integrity": "sha512-5ON+q7jCTgMp9cjpu4Jo6XbvfYwSB2Ow3kzHKfIyJfaCAOHLbdKPQqGKgfED/R5B+3TFFfe8pegYA+b423SRyA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"@babel/runtime": "^7.12.5",
|
||||
"@types/raf": "^3.4.0",
|
||||
"core-js": "^3.8.3",
|
||||
"raf": "^3.4.1",
|
||||
"regenerator-runtime": "^0.13.7",
|
||||
"rgbcolor": "^1.0.1",
|
||||
"stackblur-canvas": "^2.0.0",
|
||||
"svg-pathdata": "^6.0.3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=10.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/cfb": {
|
||||
"version": "1.2.2",
|
||||
"resolved": "https://registry.npmjs.org/cfb/-/cfb-1.2.2.tgz",
|
||||
"integrity": "sha512-KfdUZsSOw19/ObEWasvBP/Ac4reZvAGauZhs6S/gqNhXhI7cKwvlH7ulj+dOEYnca4bm4SGo8C1bTAQvnTjgQA==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"adler-32": "~1.3.0",
|
||||
"crc-32": "~1.2.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/chai": {
|
||||
"version": "5.3.3",
|
||||
"resolved": "https://registry.npmjs.org/chai/-/chai-5.3.3.tgz",
|
||||
|
|
@ -5700,6 +5977,15 @@
|
|||
"url": "https://github.com/chalk/wrap-ansi?sponsor=1"
|
||||
}
|
||||
},
|
||||
"node_modules/codepage": {
|
||||
"version": "1.15.0",
|
||||
"resolved": "https://registry.npmjs.org/codepage/-/codepage-1.15.0.tgz",
|
||||
"integrity": "sha512-3g6NUTPd/YtuuGrhMnOMRjFc+LJw/bnMp3+0r/Wcz3IXUuCosKRJvMphm5+Q+bvTVGcJJuRvVLuYba+WojaFaA==",
|
||||
"license": "Apache-2.0",
|
||||
"engines": {
|
||||
"node": ">=0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/color": {
|
||||
"version": "4.2.3",
|
||||
"resolved": "https://registry.npmjs.org/color/-/color-4.2.3.tgz",
|
||||
|
|
@ -6104,6 +6390,19 @@
|
|||
"node": ">=10"
|
||||
}
|
||||
},
|
||||
"node_modules/core-js": {
|
||||
"version": "3.45.1",
|
||||
"resolved": "https://registry.npmjs.org/core-js/-/core-js-3.45.1.tgz",
|
||||
"integrity": "sha512-L4NPsJlCfZsPeXukyzHFlg/i7IIVwHSItR0wg0FLNqYClJ4MQYTYLbC7EkjKYRLZF2iof2MUgN0EGy7MdQFChg==",
|
||||
"dev": true,
|
||||
"hasInstallScript": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/core-js"
|
||||
}
|
||||
},
|
||||
"node_modules/core-util-is": {
|
||||
"version": "1.0.3",
|
||||
"resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.3.tgz",
|
||||
|
|
@ -6130,7 +6429,6 @@
|
|||
"version": "1.2.2",
|
||||
"resolved": "https://registry.npmjs.org/crc-32/-/crc-32-1.2.2.tgz",
|
||||
"integrity": "sha512-ROmzCKrTnOwybPcJApAA6WBWij23HVfGVNKqqrZpuyZOHqK2CwHSvpGuyt/UNNvaIjEd8X5IFGp4Mh+Ie1IHJQ==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"bin": {
|
||||
"crc32": "bin/crc32.njs"
|
||||
|
|
@ -6185,6 +6483,23 @@
|
|||
"node": ">= 8"
|
||||
}
|
||||
},
|
||||
"node_modules/css-line-break": {
|
||||
"version": "2.1.0",
|
||||
"resolved": "https://registry.npmjs.org/css-line-break/-/css-line-break-2.1.0.tgz",
|
||||
"integrity": "sha512-FHcKFCZcAha3LwfVBhCQbW2nCNbkZXn7KVUJcsT5/P8YmfsVja0FMPJr0B903j/E69HUphKiV9iQArX8SDYA4w==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"utrie": "^1.0.2"
|
||||
}
|
||||
},
|
||||
"node_modules/csv-parse": {
|
||||
"version": "6.1.0",
|
||||
"resolved": "https://registry.npmjs.org/csv-parse/-/csv-parse-6.1.0.tgz",
|
||||
"integrity": "sha512-CEE+jwpgLn+MmtCpVcPtiCZpVtB6Z2OKPTr34pycYYoL7sxdOkXDdQ4lRiw6ioC0q6BLqhc6cKweCVvral8yhw==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/dargs": {
|
||||
"version": "7.0.0",
|
||||
"resolved": "https://registry.npmjs.org/dargs/-/dargs-7.0.0.tgz",
|
||||
|
|
@ -6466,6 +6781,17 @@
|
|||
"uuid": "dist/bin/uuid"
|
||||
}
|
||||
},
|
||||
"node_modules/dompurify": {
|
||||
"version": "3.2.7",
|
||||
"resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.2.7.tgz",
|
||||
"integrity": "sha512-WhL/YuveyGXJaerVlMYGWhvQswa7myDG17P7Vu65EWC05o8vfeNbvNf4d/BOvH99+ZW+LlQsc1GDKMa1vNK6dw==",
|
||||
"dev": true,
|
||||
"license": "(MPL-2.0 OR Apache-2.0)",
|
||||
"optional": true,
|
||||
"optionalDependencies": {
|
||||
"@types/trusted-types": "^2.0.7"
|
||||
}
|
||||
},
|
||||
"node_modules/dot-prop": {
|
||||
"version": "5.3.0",
|
||||
"resolved": "https://registry.npmjs.org/dot-prop/-/dot-prop-5.3.0.tgz",
|
||||
|
|
@ -7114,6 +7440,18 @@
|
|||
"license": "MIT",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/fast-png": {
|
||||
"version": "6.4.0",
|
||||
"resolved": "https://registry.npmjs.org/fast-png/-/fast-png-6.4.0.tgz",
|
||||
"integrity": "sha512-kAqZq1TlgBjZcLr5mcN6NP5Rv4V2f22z00c3g8vRrwkcqjerx7BEhPbOnWCPqaHUl2XWQBJQvOT/FQhdMT7X/Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@types/pako": "^2.0.3",
|
||||
"iobuffer": "^5.3.2",
|
||||
"pako": "^2.1.0"
|
||||
}
|
||||
},
|
||||
"node_modules/fast-xml-parser": {
|
||||
"version": "5.2.5",
|
||||
"resolved": "https://registry.npmjs.org/fast-xml-parser/-/fast-xml-parser-5.2.5.tgz",
|
||||
|
|
@ -7160,6 +7498,13 @@
|
|||
}
|
||||
}
|
||||
},
|
||||
"node_modules/fflate": {
|
||||
"version": "0.8.2",
|
||||
"resolved": "https://registry.npmjs.org/fflate/-/fflate-0.8.2.tgz",
|
||||
"integrity": "sha512-cPJU47OaAoCbg0pBvzsgpTPhmhqI5eJjh/JIu8tPj5q+T7iLvW/JAYUqmE7KOB4R1ZyEhzBaIQpQpardBF5z8A==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/figures": {
|
||||
"version": "3.2.0",
|
||||
"resolved": "https://registry.npmjs.org/figures/-/figures-3.2.0.tgz",
|
||||
|
|
@ -7302,6 +7647,15 @@
|
|||
"url": "https://github.com/sponsors/isaacs"
|
||||
}
|
||||
},
|
||||
"node_modules/frac": {
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/frac/-/frac-1.1.2.tgz",
|
||||
"integrity": "sha512-w/XBfkibaTl3YDqASwfDUqkna4Z2p9cFSr1aHDt0WoMTECnRfBOv2WArlZILlqgWlmdIlALXGpM2AOhEk5W3IA==",
|
||||
"license": "Apache-2.0",
|
||||
"engines": {
|
||||
"node": ">=0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/fs-constants": {
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz",
|
||||
|
|
@ -7805,6 +8159,21 @@
|
|||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/html2canvas": {
|
||||
"version": "1.4.1",
|
||||
"resolved": "https://registry.npmjs.org/html2canvas/-/html2canvas-1.4.1.tgz",
|
||||
"integrity": "sha512-fPU6BHNpsyIhr8yyMpTLLxAbkaK8ArIBcmZIRiBLiDhjeqvXolaEmDGmELFuX9I4xDcaKKcJl+TKZLqruBbmWA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"css-line-break": "^2.1.0",
|
||||
"text-segmentation": "^1.0.3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=8.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/iconv-lite": {
|
||||
"version": "0.6.3",
|
||||
"resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.6.3.tgz",
|
||||
|
|
@ -7927,6 +8296,13 @@
|
|||
}
|
||||
}
|
||||
},
|
||||
"node_modules/iobuffer": {
|
||||
"version": "5.4.0",
|
||||
"resolved": "https://registry.npmjs.org/iobuffer/-/iobuffer-5.4.0.tgz",
|
||||
"integrity": "sha512-DRebOWuqDvxunfkNJAlc3IzWIPD5xVxwUNbHr7xKB8E6aLJxIPfNX3CoMJghcFjpv6RWQsrcJbghtEwSPoJqMA==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/ipaddr.js": {
|
||||
"version": "2.2.0",
|
||||
"resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-2.2.0.tgz",
|
||||
|
|
@ -8348,6 +8724,24 @@
|
|||
"node": "*"
|
||||
}
|
||||
},
|
||||
"node_modules/jspdf": {
|
||||
"version": "3.0.3",
|
||||
"resolved": "https://registry.npmjs.org/jspdf/-/jspdf-3.0.3.tgz",
|
||||
"integrity": "sha512-eURjAyz5iX1H8BOYAfzvdPfIKK53V7mCpBTe7Kb16PaM8JSXEcUQNBQaiWMI8wY5RvNOPj4GccMjTlfwRBd+oQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/runtime": "^7.26.9",
|
||||
"fast-png": "^6.2.0",
|
||||
"fflate": "^0.8.1"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"canvg": "^3.0.11",
|
||||
"core-js": "^3.6.0",
|
||||
"dompurify": "^3.2.4",
|
||||
"html2canvas": "^1.0.0-rc.5"
|
||||
}
|
||||
},
|
||||
"node_modules/keyv": {
|
||||
"version": "4.5.4",
|
||||
"resolved": "https://registry.npmjs.org/keyv/-/keyv-4.5.4.tgz",
|
||||
|
|
@ -9344,6 +9738,13 @@
|
|||
"dev": true,
|
||||
"license": "BlueOak-1.0.0"
|
||||
},
|
||||
"node_modules/pako": {
|
||||
"version": "2.1.0",
|
||||
"resolved": "https://registry.npmjs.org/pako/-/pako-2.1.0.tgz",
|
||||
"integrity": "sha512-w+eufiZ1WuJYgPXbV/PO3NCMEc3xqylkKHzp8bxp1uW4qaSNQUkwmLLEc3kKsfz8lpV1F8Ht3U1Cm+9Srog2ug==",
|
||||
"dev": true,
|
||||
"license": "(MIT AND Zlib)"
|
||||
},
|
||||
"node_modules/parent-module": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/parent-module/-/parent-module-1.0.1.tgz",
|
||||
|
|
@ -9456,6 +9857,26 @@
|
|||
"node": ">= 14.16"
|
||||
}
|
||||
},
|
||||
"node_modules/pdfjs-dist": {
|
||||
"version": "4.10.38",
|
||||
"resolved": "https://registry.npmjs.org/pdfjs-dist/-/pdfjs-dist-4.10.38.tgz",
|
||||
"integrity": "sha512-/Y3fcFrXEAsMjJXeL9J8+ZG9U01LbuWaYypvDW2ycW1jL269L3js3DVBjDJ0Up9Np1uqDXsDrRihHANhZOlwdQ==",
|
||||
"license": "Apache-2.0",
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@napi-rs/canvas": "^0.1.65"
|
||||
}
|
||||
},
|
||||
"node_modules/performance-now": {
|
||||
"version": "2.1.0",
|
||||
"resolved": "https://registry.npmjs.org/performance-now/-/performance-now-2.1.0.tgz",
|
||||
"integrity": "sha512-7EAHlyLHI56VEIdK57uwHdHKIaAGbnXPiw0yWbarQZOKaKpvUIgW0jWRVLiatnM+XXlSwsanIBH/hzGMJulMow==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true
|
||||
},
|
||||
"node_modules/picocolors": {
|
||||
"version": "1.1.1",
|
||||
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
|
||||
|
|
@ -9729,6 +10150,17 @@
|
|||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/raf": {
|
||||
"version": "3.4.1",
|
||||
"resolved": "https://registry.npmjs.org/raf/-/raf-3.4.1.tgz",
|
||||
"integrity": "sha512-Sq4CW4QhwOHE8ucn6J34MqtZCeWFP2aQSmrlroYgqAV1PjStIhJXxYuTgUIfkEk7zTLjmIjLmU5q+fbD1NnOJA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"performance-now": "^2.1.0"
|
||||
}
|
||||
},
|
||||
"node_modules/randombytes": {
|
||||
"version": "2.1.0",
|
||||
"resolved": "https://registry.npmjs.org/randombytes/-/randombytes-2.1.0.tgz",
|
||||
|
|
@ -9923,6 +10355,14 @@
|
|||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/regenerator-runtime": {
|
||||
"version": "0.13.11",
|
||||
"resolved": "https://registry.npmjs.org/regenerator-runtime/-/regenerator-runtime-0.13.11.tgz",
|
||||
"integrity": "sha512-kY1AZVr2Ra+t+piVaJ4gxaFaReZVH40AKNo7UCX6W+dEwBo/2oZJzqfuN1qLq1oL45o56cPaTXELwrTh8Fpggg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true
|
||||
},
|
||||
"node_modules/require-directory": {
|
||||
"version": "2.1.1",
|
||||
"resolved": "https://registry.npmjs.org/require-directory/-/require-directory-2.1.1.tgz",
|
||||
|
|
@ -10012,6 +10452,17 @@
|
|||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/rgbcolor": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/rgbcolor/-/rgbcolor-1.0.1.tgz",
|
||||
"integrity": "sha512-9aZLIrhRaD97sgVhtJOW6ckOEh6/GnvQtdVNfdZ6s67+3/XwLS9lBcQYzEEhYVeUowN7pRzMLsyGhK2i/xvWbw==",
|
||||
"dev": true,
|
||||
"license": "MIT OR SEE LICENSE IN FEEL-FREE.md",
|
||||
"optional": true,
|
||||
"engines": {
|
||||
"node": ">= 0.8.15"
|
||||
}
|
||||
},
|
||||
"node_modules/roarr": {
|
||||
"version": "2.15.4",
|
||||
"resolved": "https://registry.npmjs.org/roarr/-/roarr-2.15.4.tgz",
|
||||
|
|
@ -10443,6 +10894,18 @@
|
|||
"integrity": "sha512-Oo+0REFV59/rz3gfJNKQiBlwfHaSESl1pcGyABQsnnIfWOFt6JNj5gCog2U6MLZ//IGYD+nA8nI+mTShREReaA==",
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/ssf": {
|
||||
"version": "0.11.2",
|
||||
"resolved": "https://registry.npmjs.org/ssf/-/ssf-0.11.2.tgz",
|
||||
"integrity": "sha512-+idbmIXoYET47hH+d7dfm2epdOMUDjqcB4648sTZ+t2JwoyBFL/insLfB/racrDmsKB3diwsDA696pZMieAC5g==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"frac": "~1.1.2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/ssh-remote-port-forward": {
|
||||
"version": "1.0.4",
|
||||
"resolved": "https://registry.npmjs.org/ssh-remote-port-forward/-/ssh-remote-port-forward-1.0.4.tgz",
|
||||
|
|
@ -10490,6 +10953,17 @@
|
|||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/stackblur-canvas": {
|
||||
"version": "2.7.0",
|
||||
"resolved": "https://registry.npmjs.org/stackblur-canvas/-/stackblur-canvas-2.7.0.tgz",
|
||||
"integrity": "sha512-yf7OENo23AGJhBriGx0QivY5JP6Y1HbrrDI6WLt6C5auYZXlQrheoY8hD4ibekFKz1HOfE48Ww8kMWMnJD/zcQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"engines": {
|
||||
"node": ">=0.1.14"
|
||||
}
|
||||
},
|
||||
"node_modules/standard-version": {
|
||||
"version": "9.5.0",
|
||||
"resolved": "https://registry.npmjs.org/standard-version/-/standard-version-9.5.0.tgz",
|
||||
|
|
@ -10866,6 +11340,17 @@
|
|||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/svg-pathdata": {
|
||||
"version": "6.0.3",
|
||||
"resolved": "https://registry.npmjs.org/svg-pathdata/-/svg-pathdata-6.0.3.tgz",
|
||||
"integrity": "sha512-qsjeeq5YjBZ5eMdFuUa4ZosMLxgr5RZ+F+Y1OrDhuOCEInRMA3x74XdBtggJcj9kOeInz0WE+LgCPDkZFlBYJw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"engines": {
|
||||
"node": ">=12.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/tar": {
|
||||
"version": "7.4.3",
|
||||
"resolved": "https://registry.npmjs.org/tar/-/tar-7.4.3.tgz",
|
||||
|
|
@ -10995,6 +11480,17 @@
|
|||
"node": ">=0.10"
|
||||
}
|
||||
},
|
||||
"node_modules/text-segmentation": {
|
||||
"version": "1.0.3",
|
||||
"resolved": "https://registry.npmjs.org/text-segmentation/-/text-segmentation-1.0.3.tgz",
|
||||
"integrity": "sha512-iOiPUo/BGnZ6+54OsWxZidGCsdU8YbE4PSpdPinp7DeMtUJNJBoJ/ouUSTJjHkh1KntHaltHl/gDs2FC4i5+Nw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"utrie": "^1.0.2"
|
||||
}
|
||||
},
|
||||
"node_modules/through": {
|
||||
"version": "2.3.8",
|
||||
"resolved": "https://registry.npmjs.org/through/-/through-2.3.8.tgz",
|
||||
|
|
@ -11261,6 +11757,17 @@
|
|||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/utrie": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/utrie/-/utrie-1.0.2.tgz",
|
||||
"integrity": "sha512-1MLa5ouZiOmQzUbjbu9VmjLzn1QLXBhwpUa7kdLUQK+KQ5KA9I1vk5U4YHe/X2Ch7PYnJfWuWT+VbuxbGwljhw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"base64-arraybuffer": "^1.0.2"
|
||||
}
|
||||
},
|
||||
"node_modules/uuid": {
|
||||
"version": "9.0.1",
|
||||
"resolved": "https://registry.npmjs.org/uuid/-/uuid-9.0.1.tgz",
|
||||
|
|
@ -11562,6 +12069,24 @@
|
|||
"url": "https://github.com/sponsors/sindresorhus"
|
||||
}
|
||||
},
|
||||
"node_modules/wmf": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/wmf/-/wmf-1.0.2.tgz",
|
||||
"integrity": "sha512-/p9K7bEh0Dj6WbXg4JG0xvLQmIadrner1bi45VMJTfnbVHsc7yIajZyoSoK60/dtVBs12Fm6WkUI5/3WAVsNMw==",
|
||||
"license": "Apache-2.0",
|
||||
"engines": {
|
||||
"node": ">=0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/word": {
|
||||
"version": "0.3.0",
|
||||
"resolved": "https://registry.npmjs.org/word/-/word-0.3.0.tgz",
|
||||
"integrity": "sha512-OELeY0Q61OXpdUfTp+oweA/vtLVg5VDOXh+3he3PNzLGG/y0oylSOC1xRVj0+l4vQ3tj/bB1HVHv1ocXkQceFA==",
|
||||
"license": "Apache-2.0",
|
||||
"engines": {
|
||||
"node": ">=0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/word-wrap": {
|
||||
"version": "1.2.5",
|
||||
"resolved": "https://registry.npmjs.org/word-wrap/-/word-wrap-1.2.5.tgz",
|
||||
|
|
@ -11706,6 +12231,27 @@
|
|||
}
|
||||
}
|
||||
},
|
||||
"node_modules/xlsx": {
|
||||
"version": "0.18.5",
|
||||
"resolved": "https://registry.npmjs.org/xlsx/-/xlsx-0.18.5.tgz",
|
||||
"integrity": "sha512-dmg3LCjBPHZnQp5/F/+nnTa+miPJxUXB6vtk42YjBBKayDNagxGEeIdWApkYPOf3Z3pm3k62Knjzp7lMeTEtFQ==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"adler-32": "~1.3.0",
|
||||
"cfb": "~1.2.1",
|
||||
"codepage": "~1.15.0",
|
||||
"crc-32": "~1.2.1",
|
||||
"ssf": "~0.11.2",
|
||||
"wmf": "~1.0.1",
|
||||
"word": "~0.3.0"
|
||||
},
|
||||
"bin": {
|
||||
"xlsx": "bin/xlsx.njs"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/xml2js": {
|
||||
"version": "0.6.2",
|
||||
"resolved": "https://registry.npmjs.org/xml2js/-/xml2js-0.6.2.tgz",
|
||||
|
|
|
|||
|
|
@ -147,6 +147,7 @@
|
|||
"@typescript-eslint/eslint-plugin": "^8.0.0",
|
||||
"@typescript-eslint/parser": "^8.0.0",
|
||||
"@vitest/coverage-v8": "^3.2.4",
|
||||
"jspdf": "^3.0.3",
|
||||
"minio": "^8.0.5",
|
||||
"standard-version": "^9.5.0",
|
||||
"testcontainers": "^11.5.1",
|
||||
|
|
@ -159,13 +160,17 @@
|
|||
"@huggingface/transformers": "^3.7.2",
|
||||
"boxen": "^8.0.1",
|
||||
"chalk": "^5.3.0",
|
||||
"chardet": "^2.0.0",
|
||||
"cli-table3": "^0.6.5",
|
||||
"commander": "^11.1.0",
|
||||
"csv-parse": "^6.1.0",
|
||||
"inquirer": "^12.9.3",
|
||||
"ora": "^8.2.0",
|
||||
"pdfjs-dist": "^4.0.379",
|
||||
"prompts": "^2.4.2",
|
||||
"uuid": "^9.0.1",
|
||||
"ws": "^8.18.3"
|
||||
"ws": "^8.18.3",
|
||||
"xlsx": "^0.18.5"
|
||||
},
|
||||
"prettier": {
|
||||
"arrowParens": "always",
|
||||
|
|
|
|||
|
|
@ -10,6 +10,47 @@ Brainy's functionality in various ways.
|
|||
|
||||
## Available Augmentations
|
||||
|
||||
### Core Augmentations
|
||||
|
||||
#### IntelligentImportAugmentation
|
||||
|
||||
Automatically detects and processes CSV, Excel, and PDF files with intelligent extraction. This augmentation is enabled by default and provides:
|
||||
|
||||
- **CSV Support**: Auto-detection of encoding, delimiters, and field types
|
||||
- **Excel Support**: Multi-sheet extraction with metadata preservation
|
||||
- **PDF Support**: Text extraction, table detection, and metadata extraction
|
||||
- **Type Inference**: Automatically infers data types (string, number, boolean, date)
|
||||
- **Neural Integration**: Seamlessly integrates with entity extraction and relationship detection
|
||||
|
||||
```javascript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy({
|
||||
intelligentImport: {
|
||||
enableCSV: true,
|
||||
enableExcel: true,
|
||||
enablePDF: true,
|
||||
maxFileSize: 100 * 1024 * 1024 // 100MB
|
||||
}
|
||||
})
|
||||
await brain.init()
|
||||
|
||||
// Import CSV with auto-detection
|
||||
await brain.import('customers.csv')
|
||||
|
||||
// Import Excel with specific sheets
|
||||
await brain.import('sales-data.xlsx', {
|
||||
excelSheets: ['Q1', 'Q2']
|
||||
})
|
||||
|
||||
// Import PDF with table extraction
|
||||
await brain.import('report.pdf', {
|
||||
pdfExtractTables: true
|
||||
})
|
||||
```
|
||||
|
||||
**See:** [Import Anything Guide](../../docs/guides/import-anything.md) | [Example](../../examples/import-excel-pdf-csv.ts)
|
||||
|
||||
### Conduit Augmentations
|
||||
|
||||
Conduit augmentations provide data synchronization between Brainy instances.
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ import { CacheAugmentation } from './cacheAugmentation.js'
|
|||
import { MetricsAugmentation } from './metricsAugmentation.js'
|
||||
import { MonitoringAugmentation } from './monitoringAugmentation.js'
|
||||
import { UniversalDisplayAugmentation } from './universalDisplayAugmentation.js'
|
||||
import { IntelligentImportAugmentation } from './intelligentImport/index.js'
|
||||
|
||||
/**
|
||||
* Create default augmentations for zero-config operation
|
||||
|
|
@ -28,10 +29,17 @@ export function createDefaultAugmentations(
|
|||
metrics?: boolean | Record<string, any>
|
||||
monitoring?: boolean | Record<string, any>
|
||||
display?: boolean | Record<string, any>
|
||||
intelligentImport?: boolean | Record<string, any>
|
||||
} = {}
|
||||
): BaseAugmentation[] {
|
||||
const augmentations: BaseAugmentation[] = []
|
||||
|
||||
// Intelligent Import augmentation (CSV, Excel, PDF)
|
||||
if (config.intelligentImport !== false) {
|
||||
const importConfig = typeof config.intelligentImport === 'object' ? config.intelligentImport : {}
|
||||
augmentations.push(new IntelligentImportAugmentation(importConfig))
|
||||
}
|
||||
|
||||
// Cache augmentation (was SearchCache)
|
||||
if (config.cache !== false) {
|
||||
const cacheConfig = typeof config.cache === 'object' ? config.cache : {}
|
||||
|
|
@ -111,5 +119,12 @@ export const AugmentationHelpers = {
|
|||
*/
|
||||
getDisplay(brain: Brainy): UniversalDisplayAugmentation | null {
|
||||
return getAugmentation<UniversalDisplayAugmentation>(brain, 'display')
|
||||
},
|
||||
|
||||
/**
|
||||
* Get intelligent import augmentation
|
||||
*/
|
||||
getIntelligentImport(brain: Brainy): IntelligentImportAugmentation | null {
|
||||
return getAugmentation<IntelligentImportAugmentation>(brain, 'intelligent-import')
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,220 @@
|
|||
/**
|
||||
* Intelligent Import Augmentation
|
||||
*
|
||||
* Automatically detects and processes CSV, Excel, and PDF files with:
|
||||
* - Format detection and routing
|
||||
* - Lazy-loaded handlers
|
||||
* - Intelligent entity and relationship extraction
|
||||
* - Integration with NeuralImport augmentation
|
||||
*/
|
||||
|
||||
import { BaseAugmentation, AugmentationContext } from '../brainyAugmentation.js'
|
||||
import { FormatHandler, IntelligentImportConfig, ProcessedData } from './types.js'
|
||||
import { CSVHandler } from './handlers/csvHandler.js'
|
||||
import { ExcelHandler } from './handlers/excelHandler.js'
|
||||
import { PDFHandler } from './handlers/pdfHandler.js'
|
||||
|
||||
export class IntelligentImportAugmentation extends BaseAugmentation {
|
||||
readonly name = 'intelligent-import'
|
||||
readonly timing = 'before' as const
|
||||
readonly metadata = {
|
||||
reads: '*' as '*',
|
||||
writes: ['_intelligentImport', '_processedFormat', '_extractedData'] as string[]
|
||||
}
|
||||
readonly operations = ['import', 'importFile', 'importFromFile', 'importFromURL', 'all'] as any[]
|
||||
readonly priority = 75 // Before NeuralImport (80), after validation
|
||||
|
||||
protected config: IntelligentImportConfig
|
||||
private handlers: Map<string, FormatHandler> = new Map()
|
||||
private initialized = false
|
||||
|
||||
constructor(config: Partial<IntelligentImportConfig> = {}) {
|
||||
super(config)
|
||||
this.config = {
|
||||
enableCSV: true,
|
||||
enableExcel: true,
|
||||
enablePDF: true,
|
||||
maxFileSize: 100 * 1024 * 1024, // 100MB default
|
||||
enableCache: true,
|
||||
cacheTTL: 24 * 60 * 60 * 1000, // 24 hours
|
||||
...config
|
||||
}
|
||||
}
|
||||
|
||||
protected async onInitialize(): Promise<void> {
|
||||
// Initialize handlers based on config
|
||||
if (this.config.enableCSV) {
|
||||
this.handlers.set('csv', new CSVHandler())
|
||||
}
|
||||
|
||||
if (this.config.enableExcel) {
|
||||
this.handlers.set('excel', new ExcelHandler())
|
||||
}
|
||||
|
||||
if (this.config.enablePDF) {
|
||||
this.handlers.set('pdf', new PDFHandler())
|
||||
}
|
||||
|
||||
this.initialized = true
|
||||
this.log(`Initialized with ${this.handlers.size} format handlers (CSV: ${this.config.enableCSV}, Excel: ${this.config.enableExcel}, PDF: ${this.config.enablePDF})`)
|
||||
}
|
||||
|
||||
async execute<T = any>(
|
||||
operation: string,
|
||||
params: any,
|
||||
next: () => Promise<T>
|
||||
): Promise<T> {
|
||||
// Only process import operations
|
||||
if (!this.shouldProcess(operation, params)) {
|
||||
return next()
|
||||
}
|
||||
|
||||
try {
|
||||
// Extract file data from params
|
||||
const fileData = this.extractFileData(params)
|
||||
if (!fileData) {
|
||||
return next()
|
||||
}
|
||||
|
||||
// Check file size limit
|
||||
if (this.config.maxFileSize && fileData.data.length > this.config.maxFileSize) {
|
||||
this.log(`File too large (${fileData.data.length} bytes), skipping intelligent import`, 'warn')
|
||||
return next()
|
||||
}
|
||||
|
||||
// Detect format and get appropriate handler
|
||||
const handler = this.detectHandler(fileData.data, fileData.filename)
|
||||
if (!handler) {
|
||||
// Not a supported format, pass through
|
||||
return next()
|
||||
}
|
||||
|
||||
this.log(`Processing ${fileData.filename || 'file'} with ${handler.format} handler`)
|
||||
|
||||
// Process the file
|
||||
const processed = await handler.process(fileData.data, {
|
||||
filename: fileData.filename,
|
||||
ext: fileData.ext,
|
||||
...this.config.csvDefaults,
|
||||
...this.config.excelDefaults,
|
||||
...this.config.pdfDefaults,
|
||||
...params.options
|
||||
})
|
||||
|
||||
// Enrich params with processed data
|
||||
params._intelligentImport = true
|
||||
params._processedFormat = processed.format
|
||||
params._extractedData = processed.data
|
||||
params._metadata = {
|
||||
...params._metadata,
|
||||
intelligentImport: processed.metadata
|
||||
}
|
||||
|
||||
// If this is an import operation, transform params to include the structured data
|
||||
if (processed.data.length > 0) {
|
||||
// Store processed data for the neural import augmentation to use
|
||||
params.data = processed.data
|
||||
params.metadata = params._metadata
|
||||
}
|
||||
|
||||
this.log(`Extracted ${processed.data.length} items from ${processed.format} file`)
|
||||
|
||||
return next()
|
||||
} catch (error) {
|
||||
this.log(`Intelligent import processing failed: ${error instanceof Error ? error.message : String(error)}`, 'warn')
|
||||
// Fall through to normal import on error
|
||||
return next()
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if we should process this operation
|
||||
*/
|
||||
private shouldProcess(operation: string, params: any): boolean {
|
||||
// Only process if we have handlers initialized
|
||||
if (!this.initialized || this.handlers.size === 0) {
|
||||
return false
|
||||
}
|
||||
|
||||
// Check operation type
|
||||
const validOps = ['import', 'importFile', 'importFromFile', 'importFromURL']
|
||||
if (!validOps.some(op => operation.includes(op))) {
|
||||
return false
|
||||
}
|
||||
|
||||
// Must have some data
|
||||
if (!params || (!params.source && !params.data && !params.filePath && !params.url)) {
|
||||
return false
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract file data from various param formats
|
||||
*/
|
||||
private extractFileData(params: any): { data: Buffer, filename?: string, ext?: string } | null {
|
||||
// From source parameter
|
||||
if (params.source) {
|
||||
if (Buffer.isBuffer(params.source)) {
|
||||
return { data: params.source, filename: params.filename }
|
||||
}
|
||||
if (typeof params.source === 'string') {
|
||||
return { data: Buffer.from(params.source), filename: params.filename }
|
||||
}
|
||||
}
|
||||
|
||||
// From data parameter
|
||||
if (params.data) {
|
||||
if (Buffer.isBuffer(params.data)) {
|
||||
return { data: params.data, filename: params.filename }
|
||||
}
|
||||
if (typeof params.data === 'string') {
|
||||
return { data: Buffer.from(params.data), filename: params.filename }
|
||||
}
|
||||
}
|
||||
|
||||
// From file path (would need to read - but that should be handled by UniversalImportAPI)
|
||||
if (params.filePath && typeof params.filePath === 'string') {
|
||||
const ext = params.filePath.split('.').pop()
|
||||
return null // File reading handled elsewhere
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect which handler can process this file
|
||||
*/
|
||||
private detectHandler(data: Buffer, filename?: string): FormatHandler | null {
|
||||
// Try each handler's canHandle method
|
||||
for (const handler of this.handlers.values()) {
|
||||
if (handler.canHandle(data) || (filename && handler.canHandle({ filename }))) {
|
||||
return handler
|
||||
}
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
|
||||
/**
|
||||
* Get handler by format name
|
||||
*/
|
||||
getHandler(format: string): FormatHandler | undefined {
|
||||
return this.handlers.get(format.toLowerCase())
|
||||
}
|
||||
|
||||
/**
|
||||
* Get all registered handlers
|
||||
*/
|
||||
getHandlers(): FormatHandler[] {
|
||||
return Array.from(this.handlers.values())
|
||||
}
|
||||
|
||||
/**
|
||||
* Get supported formats
|
||||
*/
|
||||
getSupportedFormats(): string[] {
|
||||
return Array.from(this.handlers.keys())
|
||||
}
|
||||
}
|
||||
169
src/augmentations/intelligentImport/handlers/base.ts
Normal file
169
src/augmentations/intelligentImport/handlers/base.ts
Normal file
|
|
@ -0,0 +1,169 @@
|
|||
/**
|
||||
* Base Format Handler
|
||||
* Abstract class providing common functionality for all format handlers
|
||||
*/
|
||||
|
||||
import { FormatHandler, FormatHandlerOptions, ProcessedData } from '../types.js'
|
||||
|
||||
export abstract class BaseFormatHandler implements FormatHandler {
|
||||
abstract readonly format: string
|
||||
|
||||
abstract process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData>
|
||||
|
||||
abstract canHandle(data: Buffer | string | { filename?: string, ext?: string }): boolean
|
||||
|
||||
/**
|
||||
* Detect file extension from various inputs
|
||||
*/
|
||||
protected detectExtension(data: Buffer | string | { filename?: string, ext?: string }): string | null {
|
||||
if (typeof data === 'object' && 'filename' in data && data.filename) {
|
||||
return this.getExtension(data.filename)
|
||||
}
|
||||
if (typeof data === 'object' && 'ext' in data && data.ext) {
|
||||
return data.ext.toLowerCase().replace(/^\./, '')
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract extension from filename
|
||||
*/
|
||||
protected getExtension(filename: string): string {
|
||||
const match = filename.match(/\.([^.]+)$/)
|
||||
return match ? match[1].toLowerCase() : ''
|
||||
}
|
||||
|
||||
/**
|
||||
* Infer field types from data
|
||||
* Analyzes multiple rows to determine the most appropriate type
|
||||
*/
|
||||
protected inferFieldTypes(data: Array<Record<string, any>>): Record<string, string> {
|
||||
if (data.length === 0) return {}
|
||||
|
||||
const types: Record<string, string> = {}
|
||||
const firstRow = data[0]
|
||||
const sampleSize = Math.min(10, data.length)
|
||||
|
||||
for (const key of Object.keys(firstRow)) {
|
||||
// Check first few rows to get more accurate type
|
||||
const sampleTypes = new Set<string>()
|
||||
|
||||
for (let i = 0; i < sampleSize; i++) {
|
||||
const value = data[i][key]
|
||||
const type = this.inferType(value)
|
||||
sampleTypes.add(type)
|
||||
}
|
||||
|
||||
// If we see both integer and float, use float
|
||||
if (sampleTypes.has('float') || (sampleTypes.has('integer') && sampleTypes.has('float'))) {
|
||||
types[key] = 'float'
|
||||
} else if (sampleTypes.has('integer')) {
|
||||
types[key] = 'integer'
|
||||
} else if (sampleTypes.has('date')) {
|
||||
types[key] = 'date'
|
||||
} else if (sampleTypes.has('boolean')) {
|
||||
types[key] = 'boolean'
|
||||
} else {
|
||||
types[key] = 'string'
|
||||
}
|
||||
}
|
||||
|
||||
return types
|
||||
}
|
||||
|
||||
/**
|
||||
* Infer type of a single value
|
||||
*/
|
||||
protected inferType(value: any): string {
|
||||
if (value === null || value === undefined || value === '') return 'string'
|
||||
|
||||
if (typeof value === 'number') return 'number'
|
||||
if (typeof value === 'boolean') return 'boolean'
|
||||
|
||||
if (typeof value === 'string') {
|
||||
// Check if it's a number
|
||||
if (/^-?\d+$/.test(value)) return 'integer'
|
||||
if (/^-?\d+\.\d+$/.test(value)) return 'float'
|
||||
|
||||
// Check if it's a date
|
||||
if (this.isDateString(value)) return 'date'
|
||||
|
||||
// Check if it's a boolean
|
||||
if (/^(true|false|yes|no|y|n)$/i.test(value)) return 'boolean'
|
||||
}
|
||||
|
||||
return 'string'
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if string looks like a date
|
||||
*/
|
||||
protected isDateString(value: string): boolean {
|
||||
// ISO 8601
|
||||
if (/^\d{4}-\d{2}-\d{2}/.test(value)) return true
|
||||
|
||||
// Common date formats
|
||||
if (/^\d{1,2}\/\d{1,2}\/\d{2,4}$/.test(value)) return true
|
||||
if (/^\d{1,2}-\d{1,2}-\d{2,4}$/.test(value)) return true
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
/**
|
||||
* Sanitize field names for use as object keys
|
||||
*/
|
||||
protected sanitizeFieldName(name: string): string {
|
||||
return name
|
||||
.trim()
|
||||
.replace(/[^a-zA-Z0-9_\s-]/g, '')
|
||||
.replace(/\s+/g, '_')
|
||||
.replace(/-+/g, '_')
|
||||
.replace(/_+/g, '_')
|
||||
.replace(/^_|_$/g, '')
|
||||
|| 'field'
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert value to appropriate type
|
||||
*/
|
||||
protected convertValue(value: any, type: string): any {
|
||||
if (value === null || value === undefined || value === '') return null
|
||||
|
||||
switch (type) {
|
||||
case 'integer':
|
||||
return parseInt(String(value), 10)
|
||||
|
||||
case 'float':
|
||||
case 'number':
|
||||
return parseFloat(String(value))
|
||||
|
||||
case 'boolean':
|
||||
if (typeof value === 'boolean') return value
|
||||
const str = String(value).toLowerCase()
|
||||
return ['true', 'yes', 'y', '1'].includes(str)
|
||||
|
||||
case 'date':
|
||||
return new Date(value)
|
||||
|
||||
default:
|
||||
return value
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Create metadata object with common fields
|
||||
*/
|
||||
protected createMetadata(
|
||||
rowCount: number,
|
||||
fields: string[],
|
||||
processingTime: number,
|
||||
extra: Record<string, any> = {}
|
||||
): ProcessedData['metadata'] {
|
||||
return {
|
||||
rowCount,
|
||||
fields,
|
||||
processingTime,
|
||||
...extra
|
||||
}
|
||||
}
|
||||
}
|
||||
210
src/augmentations/intelligentImport/handlers/csvHandler.ts
Normal file
210
src/augmentations/intelligentImport/handlers/csvHandler.ts
Normal file
|
|
@ -0,0 +1,210 @@
|
|||
/**
|
||||
* CSV Format Handler
|
||||
* Handles CSV files with:
|
||||
* - Automatic encoding detection
|
||||
* - Automatic delimiter detection
|
||||
* - Streaming for large files
|
||||
* - Type inference
|
||||
*/
|
||||
|
||||
import { parse } from 'csv-parse/sync'
|
||||
import { detect as detectEncoding } from 'chardet'
|
||||
import { BaseFormatHandler } from './base.js'
|
||||
import { FormatHandlerOptions, ProcessedData } from '../types.js'
|
||||
|
||||
export class CSVHandler extends BaseFormatHandler {
|
||||
readonly format = 'csv'
|
||||
|
||||
canHandle(data: Buffer | string | { filename?: string, ext?: string }): boolean {
|
||||
const ext = this.detectExtension(data)
|
||||
if (ext === 'csv' || ext === 'tsv' || ext === 'txt') return true
|
||||
|
||||
// Check content if it's a buffer
|
||||
if (Buffer.isBuffer(data)) {
|
||||
const sample = data.slice(0, 1024).toString('utf-8')
|
||||
return this.looksLikeCSV(sample)
|
||||
}
|
||||
|
||||
if (typeof data === 'string') {
|
||||
return this.looksLikeCSV(data.slice(0, 1024))
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
|
||||
const startTime = Date.now()
|
||||
|
||||
// Convert to buffer if string
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8')
|
||||
|
||||
// Detect encoding
|
||||
const detectedEncoding = options.encoding || this.detectEncodingSafe(buffer)
|
||||
const text = buffer.toString(detectedEncoding as BufferEncoding)
|
||||
|
||||
// Detect delimiter if not specified
|
||||
const delimiter = options.csvDelimiter || this.detectDelimiter(text)
|
||||
|
||||
// Parse CSV
|
||||
const hasHeaders = options.csvHeaders !== false
|
||||
const maxRows = options.maxRows
|
||||
|
||||
try {
|
||||
const records = parse(text, {
|
||||
columns: hasHeaders,
|
||||
skip_empty_lines: true,
|
||||
trim: true,
|
||||
delimiter,
|
||||
relax_column_count: true,
|
||||
to: maxRows,
|
||||
cast: false // We'll do type inference ourselves
|
||||
})
|
||||
|
||||
// Convert to array of objects
|
||||
const data = Array.isArray(records) ? records : [records]
|
||||
|
||||
// Infer types and convert values
|
||||
const fields = data.length > 0 ? Object.keys(data[0]) : []
|
||||
const types = this.inferFieldTypes(data)
|
||||
|
||||
const convertedData = data.map(row => {
|
||||
const converted: Record<string, any> = {}
|
||||
for (const [key, value] of Object.entries(row)) {
|
||||
converted[key] = this.convertValue(value, types[key] || 'string')
|
||||
}
|
||||
return converted
|
||||
})
|
||||
|
||||
const processingTime = Date.now() - startTime
|
||||
|
||||
return {
|
||||
format: this.format,
|
||||
data: convertedData,
|
||||
metadata: this.createMetadata(
|
||||
convertedData.length,
|
||||
fields,
|
||||
processingTime,
|
||||
{
|
||||
encoding: detectedEncoding,
|
||||
delimiter,
|
||||
hasHeaders,
|
||||
types
|
||||
}
|
||||
),
|
||||
filename: options.filename
|
||||
}
|
||||
} catch (error) {
|
||||
throw new Error(`CSV parsing failed: ${error instanceof Error ? error.message : String(error)}`)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if text looks like CSV
|
||||
*/
|
||||
private looksLikeCSV(text: string): boolean {
|
||||
const lines = text.split('\n').filter(l => l.trim())
|
||||
if (lines.length < 2) return false
|
||||
|
||||
// Check for common delimiters
|
||||
const delimiters = [',', ';', '\t', '|']
|
||||
for (const delimiter of delimiters) {
|
||||
const firstCount = (lines[0].match(new RegExp(`\\${delimiter}`, 'g')) || []).length
|
||||
if (firstCount === 0) continue
|
||||
|
||||
const secondCount = (lines[1].match(new RegExp(`\\${delimiter}`, 'g')) || []).length
|
||||
if (firstCount === secondCount) return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect CSV delimiter
|
||||
*/
|
||||
private detectDelimiter(text: string): string {
|
||||
const sample = text.split('\n').slice(0, 10).join('\n')
|
||||
const delimiters = [',', ';', '\t', '|']
|
||||
const counts: Record<string, number> = {}
|
||||
|
||||
for (const delimiter of delimiters) {
|
||||
const lines = sample.split('\n').filter(l => l.trim())
|
||||
if (lines.length < 2) continue
|
||||
|
||||
// Count delimiter in first line
|
||||
const firstCount = (lines[0].match(new RegExp(`\\${delimiter}`, 'g')) || []).length
|
||||
if (firstCount === 0) continue
|
||||
|
||||
// Check if count is consistent across lines
|
||||
let consistent = true
|
||||
for (let i = 1; i < Math.min(5, lines.length); i++) {
|
||||
const count = (lines[i].match(new RegExp(`\\${delimiter}`, 'g')) || []).length
|
||||
if (count !== firstCount) {
|
||||
consistent = false
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if (consistent) {
|
||||
counts[delimiter] = firstCount
|
||||
}
|
||||
}
|
||||
|
||||
// Return delimiter with highest count
|
||||
const best = Object.entries(counts).sort((a, b) => b[1] - a[1])[0]
|
||||
return best ? best[0] : ','
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect encoding safely (with fallback)
|
||||
*/
|
||||
private detectEncodingSafe(buffer: Buffer): string {
|
||||
try {
|
||||
const detected = detectEncoding(buffer)
|
||||
if (!detected) return 'utf-8'
|
||||
|
||||
// Normalize encoding to Node.js-supported names
|
||||
return this.normalizeEncoding(detected)
|
||||
} catch {
|
||||
return 'utf-8'
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize encoding names to Node.js-supported encodings
|
||||
*/
|
||||
private normalizeEncoding(encoding: string): string {
|
||||
const normalized = encoding.toLowerCase().replace(/[_-]/g, '')
|
||||
|
||||
// Map common encodings to Node.js names
|
||||
const mappings: Record<string, string> = {
|
||||
'iso88591': 'latin1',
|
||||
'iso88592': 'latin1',
|
||||
'iso88593': 'latin1',
|
||||
'iso88594': 'latin1',
|
||||
'iso88595': 'latin1',
|
||||
'iso88596': 'latin1',
|
||||
'iso88597': 'latin1',
|
||||
'iso88598': 'latin1',
|
||||
'iso88599': 'latin1',
|
||||
'iso885910': 'latin1',
|
||||
'iso885913': 'latin1',
|
||||
'iso885914': 'latin1',
|
||||
'iso885915': 'latin1',
|
||||
'iso885916': 'latin1',
|
||||
'usascii': 'ascii',
|
||||
'utf8': 'utf8',
|
||||
'utf16le': 'utf16le',
|
||||
'utf16be': 'utf16le',
|
||||
'windows1252': 'latin1',
|
||||
'windows1251': 'utf8', // Cyrillic - best effort
|
||||
'big5': 'utf8', // Chinese - best effort
|
||||
'gbk': 'utf8', // Chinese - best effort
|
||||
'gb2312': 'utf8', // Chinese - best effort
|
||||
'shiftjis': 'utf8', // Japanese - best effort
|
||||
'eucjp': 'utf8', // Japanese - best effort
|
||||
'euckr': 'utf8' // Korean - best effort
|
||||
}
|
||||
|
||||
return mappings[normalized] || 'utf8'
|
||||
}
|
||||
}
|
||||
176
src/augmentations/intelligentImport/handlers/excelHandler.ts
Normal file
176
src/augmentations/intelligentImport/handlers/excelHandler.ts
Normal file
|
|
@ -0,0 +1,176 @@
|
|||
/**
|
||||
* Excel Format Handler
|
||||
* Handles Excel files (.xlsx, .xls, .xlsb) with:
|
||||
* - Multi-sheet extraction
|
||||
* - Type inference
|
||||
* - Formula evaluation
|
||||
* - Metadata extraction
|
||||
*/
|
||||
|
||||
import * as XLSX from 'xlsx'
|
||||
import { BaseFormatHandler } from './base.js'
|
||||
import { FormatHandlerOptions, ProcessedData } from '../types.js'
|
||||
|
||||
export class ExcelHandler extends BaseFormatHandler {
|
||||
readonly format = 'excel'
|
||||
|
||||
canHandle(data: Buffer | string | { filename?: string, ext?: string }): boolean {
|
||||
const ext = this.detectExtension(data)
|
||||
return ['xlsx', 'xls', 'xlsb', 'xlsm', 'xlt', 'xltx', 'xltm'].includes(ext || '')
|
||||
}
|
||||
|
||||
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
|
||||
const startTime = Date.now()
|
||||
|
||||
// Convert to buffer if string (though Excel should always be binary)
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'binary')
|
||||
|
||||
try {
|
||||
// Read workbook
|
||||
const workbook = XLSX.read(buffer, {
|
||||
type: 'buffer',
|
||||
cellDates: true,
|
||||
cellNF: true,
|
||||
cellStyles: true
|
||||
})
|
||||
|
||||
// Determine which sheets to process
|
||||
const sheetsToProcess = this.getSheetsToProcess(workbook, options)
|
||||
|
||||
// Extract data from sheets
|
||||
const allData: Array<Record<string, any>> = []
|
||||
const sheetMetadata: Record<string, any> = {}
|
||||
|
||||
for (const sheetName of sheetsToProcess) {
|
||||
const sheet = workbook.Sheets[sheetName]
|
||||
if (!sheet) continue
|
||||
|
||||
// Convert sheet to JSON with headers
|
||||
const sheetData = XLSX.utils.sheet_to_json(sheet, {
|
||||
header: 1, // Get as array of arrays first
|
||||
defval: null,
|
||||
blankrows: false,
|
||||
raw: false // Convert to formatted strings
|
||||
}) as any[][]
|
||||
|
||||
if (sheetData.length === 0) continue
|
||||
|
||||
// First row is headers
|
||||
const headers = sheetData[0].map((h: any) =>
|
||||
this.sanitizeFieldName(String(h || ''))
|
||||
)
|
||||
|
||||
// Skip if no headers
|
||||
if (headers.length === 0) continue
|
||||
|
||||
// Convert rows to objects
|
||||
for (let i = 1; i < sheetData.length; i++) {
|
||||
const row = sheetData[i]
|
||||
const rowObj: Record<string, any> = {}
|
||||
|
||||
// Add sheet name to each row
|
||||
rowObj._sheet = sheetName
|
||||
|
||||
for (let j = 0; j < headers.length; j++) {
|
||||
const header = headers[j]
|
||||
let value = row[j]
|
||||
|
||||
// Convert Excel dates
|
||||
if (value && typeof value === 'number' && this.isExcelDate(value)) {
|
||||
value = this.excelDateToJSDate(value)
|
||||
}
|
||||
|
||||
rowObj[header] = value === undefined ? null : value
|
||||
}
|
||||
|
||||
allData.push(rowObj)
|
||||
}
|
||||
|
||||
// Store sheet metadata
|
||||
sheetMetadata[sheetName] = {
|
||||
rowCount: sheetData.length - 1, // Exclude header row
|
||||
columnCount: headers.length,
|
||||
headers
|
||||
}
|
||||
}
|
||||
|
||||
// Infer types (excluding _sheet field)
|
||||
const fields = allData.length > 0 ? Object.keys(allData[0]).filter(k => k !== '_sheet') : []
|
||||
const types = this.inferFieldTypes(allData)
|
||||
|
||||
// Convert values to appropriate types
|
||||
const convertedData = allData.map(row => {
|
||||
const converted: Record<string, any> = {}
|
||||
for (const [key, value] of Object.entries(row)) {
|
||||
if (key === '_sheet') {
|
||||
converted[key] = value
|
||||
} else {
|
||||
converted[key] = this.convertValue(value, types[key] || 'string')
|
||||
}
|
||||
}
|
||||
return converted
|
||||
})
|
||||
|
||||
const processingTime = Date.now() - startTime
|
||||
|
||||
return {
|
||||
format: this.format,
|
||||
data: convertedData,
|
||||
metadata: this.createMetadata(
|
||||
convertedData.length,
|
||||
fields,
|
||||
processingTime,
|
||||
{
|
||||
sheets: sheetsToProcess,
|
||||
sheetCount: sheetsToProcess.length,
|
||||
sheetMetadata,
|
||||
types,
|
||||
workbookInfo: {
|
||||
sheetNames: workbook.SheetNames,
|
||||
properties: workbook.Props || {}
|
||||
}
|
||||
}
|
||||
),
|
||||
filename: options.filename
|
||||
}
|
||||
} catch (error) {
|
||||
throw new Error(`Excel parsing failed: ${error instanceof Error ? error.message : String(error)}`)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Determine which sheets to process
|
||||
*/
|
||||
private getSheetsToProcess(workbook: XLSX.WorkBook, options: FormatHandlerOptions): string[] {
|
||||
const allSheets = workbook.SheetNames
|
||||
|
||||
// If specific sheets requested
|
||||
if (options.excelSheets && options.excelSheets !== 'all') {
|
||||
return options.excelSheets.filter(name => allSheets.includes(name))
|
||||
}
|
||||
|
||||
// Otherwise process all sheets
|
||||
return allSheets
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if a number is likely an Excel date
|
||||
* Excel stores dates as days since 1900-01-01
|
||||
*/
|
||||
private isExcelDate(value: number): boolean {
|
||||
// Excel dates are typically between 1 and 60000 (1900 to 2064)
|
||||
// This is a heuristic - not perfect but catches most cases
|
||||
return value > 0 && value < 100000 && Number.isInteger(value)
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert Excel date (days since 1900-01-01) to JS Date
|
||||
*/
|
||||
private excelDateToJSDate(excelDate: number): Date {
|
||||
// Excel's epoch is 1900-01-01, but there's a bug where it thinks 1900 is a leap year
|
||||
// So dates before March 1, 1900 are off by one day
|
||||
const epoch = new Date(1899, 11, 30) // Dec 30, 1899
|
||||
const msPerDay = 24 * 60 * 60 * 1000
|
||||
return new Date(epoch.getTime() + excelDate * msPerDay)
|
||||
}
|
||||
}
|
||||
287
src/augmentations/intelligentImport/handlers/pdfHandler.ts
Normal file
287
src/augmentations/intelligentImport/handlers/pdfHandler.ts
Normal file
|
|
@ -0,0 +1,287 @@
|
|||
/**
|
||||
* PDF Format Handler
|
||||
* Handles PDF files with:
|
||||
* - Text extraction with layout preservation
|
||||
* - Table detection and extraction
|
||||
* - Metadata extraction (author, dates, etc.)
|
||||
* - Page-by-page processing
|
||||
*/
|
||||
|
||||
import * as pdfjsLib from 'pdfjs-dist/legacy/build/pdf.mjs'
|
||||
import { BaseFormatHandler } from './base.js'
|
||||
import { FormatHandlerOptions, ProcessedData } from '../types.js'
|
||||
|
||||
// Use built-in worker for Node.js environments
|
||||
// In production, this can be customized via options
|
||||
const initializeWorker = () => {
|
||||
if (typeof pdfjsLib.GlobalWorkerOptions.workerSrc === 'undefined' ||
|
||||
pdfjsLib.GlobalWorkerOptions.workerSrc === '') {
|
||||
// Use a data URL to avoid file system dependencies
|
||||
// This tells pdfjs to use the built-in fallback worker
|
||||
try {
|
||||
pdfjsLib.GlobalWorkerOptions.workerSrc = 'data:,'
|
||||
} catch {
|
||||
// Ignore if already set or in incompatible environment
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
initializeWorker()
|
||||
|
||||
export class PDFHandler extends BaseFormatHandler {
|
||||
readonly format = 'pdf'
|
||||
|
||||
canHandle(data: Buffer | string | { filename?: string, ext?: string }): boolean {
|
||||
const ext = this.detectExtension(data)
|
||||
if (ext === 'pdf') return true
|
||||
|
||||
// Check for PDF magic bytes
|
||||
if (Buffer.isBuffer(data)) {
|
||||
const header = data.slice(0, 5).toString('ascii')
|
||||
return header === '%PDF-'
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
|
||||
const startTime = Date.now()
|
||||
|
||||
// Convert to buffer
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'binary')
|
||||
|
||||
try {
|
||||
// Load PDF document
|
||||
const loadingTask = pdfjsLib.getDocument({
|
||||
data: new Uint8Array(buffer),
|
||||
useSystemFonts: true,
|
||||
standardFontDataUrl: undefined
|
||||
})
|
||||
|
||||
const pdfDoc = await loadingTask.promise
|
||||
|
||||
// Extract metadata
|
||||
const metadata = await pdfDoc.getMetadata()
|
||||
const numPages = pdfDoc.numPages
|
||||
|
||||
// Extract text and structure from all pages
|
||||
const allData: Array<Record<string, any>> = []
|
||||
let totalTextLength = 0
|
||||
let detectedTables = 0
|
||||
|
||||
for (let pageNum = 1; pageNum <= numPages; pageNum++) {
|
||||
const page = await pdfDoc.getPage(pageNum)
|
||||
const textContent = await page.getTextContent()
|
||||
|
||||
// Extract text items with positions
|
||||
const textItems = textContent.items.map((item: any) => ({
|
||||
text: item.str,
|
||||
x: item.transform[4],
|
||||
y: item.transform[5],
|
||||
width: item.width,
|
||||
height: item.height
|
||||
}))
|
||||
|
||||
// Combine text items into lines (group by similar Y position)
|
||||
const lines = this.groupIntoLines(textItems)
|
||||
|
||||
// Detect tables if requested
|
||||
if (options.pdfExtractTables !== false) {
|
||||
const tables = this.detectTables(lines)
|
||||
if (tables.length > 0) {
|
||||
detectedTables += tables.length
|
||||
for (const table of tables) {
|
||||
allData.push(...table.rows)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Extract paragraphs from non-table lines
|
||||
const paragraphs = this.extractParagraphs(lines)
|
||||
for (let i = 0; i < paragraphs.length; i++) {
|
||||
const text = paragraphs[i].trim()
|
||||
if (text.length > 0) {
|
||||
totalTextLength += text.length
|
||||
allData.push({
|
||||
_page: pageNum,
|
||||
_type: 'paragraph',
|
||||
_index: i,
|
||||
text
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const processingTime = Date.now() - startTime
|
||||
|
||||
// Get all unique fields (excluding metadata fields)
|
||||
const fields = allData.length > 0
|
||||
? Object.keys(allData[0]).filter(k => !k.startsWith('_'))
|
||||
: []
|
||||
|
||||
return {
|
||||
format: this.format,
|
||||
data: allData,
|
||||
metadata: this.createMetadata(
|
||||
allData.length,
|
||||
fields,
|
||||
processingTime,
|
||||
{
|
||||
pageCount: numPages,
|
||||
textLength: totalTextLength,
|
||||
tableCount: detectedTables,
|
||||
pdfMetadata: {
|
||||
title: (metadata.info as any)?.Title || null,
|
||||
author: (metadata.info as any)?.Author || null,
|
||||
subject: (metadata.info as any)?.Subject || null,
|
||||
creator: (metadata.info as any)?.Creator || null,
|
||||
producer: (metadata.info as any)?.Producer || null,
|
||||
creationDate: (metadata.info as any)?.CreationDate || null,
|
||||
modificationDate: (metadata.info as any)?.ModDate || null
|
||||
}
|
||||
}
|
||||
),
|
||||
filename: options.filename
|
||||
}
|
||||
} catch (error) {
|
||||
throw new Error(`PDF parsing failed: ${error instanceof Error ? error.message : String(error)}`)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Group text items into lines based on Y position
|
||||
*/
|
||||
private groupIntoLines(items: Array<{ text: string, x: number, y: number, width: number, height: number }>): Array<Array<{ text: string, x: number }>> {
|
||||
if (items.length === 0) return []
|
||||
|
||||
// Sort by Y position (descending, since PDF coordinates go bottom-up)
|
||||
const sorted = [...items].sort((a, b) => b.y - a.y)
|
||||
|
||||
const lines: Array<Array<{ text: string, x: number }>> = []
|
||||
let currentLine: Array<{ text: string, x: number }> = []
|
||||
let currentY = sorted[0].y
|
||||
|
||||
for (const item of sorted) {
|
||||
// If Y position differs by more than half the height, it's a new line
|
||||
if (Math.abs(item.y - currentY) > (item.height / 2)) {
|
||||
if (currentLine.length > 0) {
|
||||
// Sort line items by X position
|
||||
currentLine.sort((a, b) => a.x - b.x)
|
||||
lines.push(currentLine)
|
||||
}
|
||||
currentLine = []
|
||||
currentY = item.y
|
||||
}
|
||||
|
||||
if (item.text.trim()) {
|
||||
currentLine.push({ text: item.text, x: item.x })
|
||||
}
|
||||
}
|
||||
|
||||
// Add last line
|
||||
if (currentLine.length > 0) {
|
||||
currentLine.sort((a, b) => a.x - b.x)
|
||||
lines.push(currentLine)
|
||||
}
|
||||
|
||||
return lines
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect tables from lines
|
||||
* Tables are detected when multiple consecutive lines have similar structure
|
||||
*/
|
||||
private detectTables(lines: Array<Array<{ text: string, x: number }>>): Array<{ rows: Array<Record<string, any>> }> {
|
||||
const tables: Array<{ rows: Array<Record<string, any>> }> = []
|
||||
let potentialTable: Array<Array<{ text: string, x: number }>> = []
|
||||
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const line = lines[i]
|
||||
|
||||
// A line with multiple items could be part of a table
|
||||
if (line.length >= 2) {
|
||||
potentialTable.push(line)
|
||||
} else {
|
||||
// End of potential table
|
||||
if (potentialTable.length >= 3) { // Need at least header + 2 rows
|
||||
const table = this.parseTable(potentialTable)
|
||||
if (table) {
|
||||
tables.push(table)
|
||||
}
|
||||
}
|
||||
potentialTable = []
|
||||
}
|
||||
}
|
||||
|
||||
// Check last potential table
|
||||
if (potentialTable.length >= 3) {
|
||||
const table = this.parseTable(potentialTable)
|
||||
if (table) {
|
||||
tables.push(table)
|
||||
}
|
||||
}
|
||||
|
||||
return tables
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a potential table into structured rows
|
||||
*/
|
||||
private parseTable(lines: Array<Array<{ text: string, x: number }>>): { rows: Array<Record<string, any>> } | null {
|
||||
if (lines.length < 2) return null
|
||||
|
||||
// First line is headers
|
||||
const headerLine = lines[0]
|
||||
const headers = headerLine.map(item => this.sanitizeFieldName(item.text))
|
||||
|
||||
// Remaining lines are data
|
||||
const rows: Array<Record<string, any>> = []
|
||||
|
||||
for (let i = 1; i < lines.length; i++) {
|
||||
const line = lines[i]
|
||||
const row: Record<string, any> = { _type: 'table_row' }
|
||||
|
||||
// Match each item to closest header by X position
|
||||
for (let j = 0; j < line.length && j < headers.length; j++) {
|
||||
const header = headers[j]
|
||||
const value = line[j].text.trim()
|
||||
row[header] = value || null
|
||||
}
|
||||
|
||||
if (Object.keys(row).length > 1) { // More than just _type
|
||||
rows.push(row)
|
||||
}
|
||||
}
|
||||
|
||||
return rows.length > 0 ? { rows } : null
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract paragraphs from lines
|
||||
*/
|
||||
private extractParagraphs(lines: Array<Array<{ text: string, x: number }>>): string[] {
|
||||
const paragraphs: string[] = []
|
||||
let currentParagraph: string[] = []
|
||||
|
||||
for (const line of lines) {
|
||||
const lineText = line.map(item => item.text).join(' ').trim()
|
||||
|
||||
if (lineText.length === 0) {
|
||||
// Empty line - end paragraph
|
||||
if (currentParagraph.length > 0) {
|
||||
paragraphs.push(currentParagraph.join(' '))
|
||||
currentParagraph = []
|
||||
}
|
||||
} else {
|
||||
currentParagraph.push(lineText)
|
||||
}
|
||||
}
|
||||
|
||||
// Add last paragraph
|
||||
if (currentParagraph.length > 0) {
|
||||
paragraphs.push(currentParagraph.join(' '))
|
||||
}
|
||||
|
||||
return paragraphs
|
||||
}
|
||||
}
|
||||
15
src/augmentations/intelligentImport/index.ts
Normal file
15
src/augmentations/intelligentImport/index.ts
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
/**
|
||||
* Intelligent Import Module
|
||||
* Exports main augmentation and types
|
||||
*/
|
||||
|
||||
export { IntelligentImportAugmentation } from './IntelligentImportAugmentation.js'
|
||||
export type {
|
||||
FormatHandler,
|
||||
FormatHandlerOptions,
|
||||
ProcessedData,
|
||||
IntelligentImportConfig
|
||||
} from './types.js'
|
||||
export { CSVHandler } from './handlers/csvHandler.js'
|
||||
export { ExcelHandler } from './handlers/excelHandler.js'
|
||||
export { PDFHandler } from './handlers/pdfHandler.js'
|
||||
148
src/augmentations/intelligentImport/types.ts
Normal file
148
src/augmentations/intelligentImport/types.ts
Normal file
|
|
@ -0,0 +1,148 @@
|
|||
/**
|
||||
* Types for Intelligent Import Augmentation
|
||||
* Handles Excel, PDF, and CSV import with intelligent extraction
|
||||
*/
|
||||
|
||||
export interface FormatHandler {
|
||||
/**
|
||||
* Format name (e.g., 'csv', 'xlsx', 'pdf')
|
||||
*/
|
||||
readonly format: string
|
||||
|
||||
/**
|
||||
* Process raw data into structured format
|
||||
* @param data Raw file data (Buffer or string)
|
||||
* @param options Format-specific options
|
||||
* @returns Structured data ready for entity extraction
|
||||
*/
|
||||
process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData>
|
||||
|
||||
/**
|
||||
* Detect if this handler can process the given data
|
||||
* @param data Raw data or filename
|
||||
* @returns true if handler supports this format
|
||||
*/
|
||||
canHandle(data: Buffer | string | { filename?: string, ext?: string }): boolean
|
||||
}
|
||||
|
||||
export interface FormatHandlerOptions {
|
||||
/** Source filename (for extension detection) */
|
||||
filename?: string
|
||||
|
||||
/** File extension (if known) */
|
||||
ext?: string
|
||||
|
||||
/** Encoding (auto-detected if not specified) */
|
||||
encoding?: string
|
||||
|
||||
/** CSV-specific: delimiter character */
|
||||
csvDelimiter?: string
|
||||
|
||||
/** CSV-specific: whether first row is headers */
|
||||
csvHeaders?: boolean
|
||||
|
||||
/** Excel-specific: sheet names to extract (or 'all') */
|
||||
excelSheets?: string[] | 'all'
|
||||
|
||||
/** Excel-specific: whether to evaluate formulas */
|
||||
excelEvaluateFormulas?: boolean
|
||||
|
||||
/** PDF-specific: whether to extract tables */
|
||||
pdfExtractTables?: boolean
|
||||
|
||||
/** PDF-specific: whether to preserve layout */
|
||||
pdfPreserveLayout?: boolean
|
||||
|
||||
/** Maximum rows to process (for large files) */
|
||||
maxRows?: number
|
||||
|
||||
/** Whether to stream large files */
|
||||
streaming?: boolean
|
||||
}
|
||||
|
||||
export interface ProcessedData {
|
||||
/** Format that was processed */
|
||||
format: string
|
||||
|
||||
/** Structured data (array of objects) */
|
||||
data: Array<Record<string, any>>
|
||||
|
||||
/** Metadata about the processed data */
|
||||
metadata: {
|
||||
/** Number of rows/entities extracted */
|
||||
rowCount: number
|
||||
|
||||
/** Column/field names */
|
||||
fields: string[]
|
||||
|
||||
/** Detected encoding (for text formats) */
|
||||
encoding?: string
|
||||
|
||||
/** Excel: sheet names */
|
||||
sheets?: string[]
|
||||
|
||||
/** PDF: page count */
|
||||
pageCount?: number
|
||||
|
||||
/** PDF: extracted text length */
|
||||
textLength?: number
|
||||
|
||||
/** PDF: number of tables detected */
|
||||
tableCount?: number
|
||||
|
||||
/** Processing time in milliseconds */
|
||||
processingTime: number
|
||||
|
||||
/** Any warnings during processing */
|
||||
warnings?: string[]
|
||||
|
||||
/** Format-specific metadata */
|
||||
[key: string]: any
|
||||
}
|
||||
|
||||
/** Original filename (if available) */
|
||||
filename?: string
|
||||
}
|
||||
|
||||
export interface HandlerRegistry {
|
||||
/** Registered handlers by format extension */
|
||||
handlers: Map<string, () => Promise<FormatHandler>>
|
||||
|
||||
/** Loaded handler instances (lazy-loaded) */
|
||||
loaded: Map<string, FormatHandler>
|
||||
|
||||
/** Register a new handler */
|
||||
register(extensions: string[], loader: () => Promise<FormatHandler>): void
|
||||
|
||||
/** Get handler for a file/format */
|
||||
getHandler(filenameOrExt: string): Promise<FormatHandler | null>
|
||||
}
|
||||
|
||||
export interface IntelligentImportConfig {
|
||||
/** Enable CSV handler */
|
||||
enableCSV: boolean
|
||||
|
||||
/** Enable Excel handler */
|
||||
enableExcel: boolean
|
||||
|
||||
/** Enable PDF handler */
|
||||
enablePDF: boolean
|
||||
|
||||
/** Default options for CSV */
|
||||
csvDefaults?: Partial<FormatHandlerOptions>
|
||||
|
||||
/** Default options for Excel */
|
||||
excelDefaults?: Partial<FormatHandlerOptions>
|
||||
|
||||
/** Default options for PDF */
|
||||
pdfDefaults?: Partial<FormatHandlerOptions>
|
||||
|
||||
/** Maximum file size to process (bytes) */
|
||||
maxFileSize?: number
|
||||
|
||||
/** Enable caching of processed data */
|
||||
enableCache?: boolean
|
||||
|
||||
/** Cache TTL in milliseconds */
|
||||
cacheTTL?: number
|
||||
}
|
||||
281
tests/augmentations/intelligentImport/csvHandler.unit.test.ts
Normal file
281
tests/augmentations/intelligentImport/csvHandler.unit.test.ts
Normal file
|
|
@ -0,0 +1,281 @@
|
|||
/**
|
||||
* CSV Handler Tests
|
||||
* Comprehensive tests for CSV parsing, encoding detection, and type inference
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll } from 'vitest'
|
||||
import { CSVHandler } from '../../../src/augmentations/intelligentImport/handlers/csvHandler.js'
|
||||
import { promises as fs } from 'fs'
|
||||
import * as path from 'path'
|
||||
|
||||
describe('CSVHandler', () => {
|
||||
let handler: CSVHandler
|
||||
const fixturesPath = path.join(process.cwd(), 'tests/fixtures/import')
|
||||
|
||||
beforeAll(() => {
|
||||
handler = new CSVHandler()
|
||||
})
|
||||
|
||||
describe('canHandle', () => {
|
||||
it('should handle .csv extension', () => {
|
||||
expect(handler.canHandle({ filename: 'data.csv' })).toBe(true)
|
||||
})
|
||||
|
||||
it('should handle .tsv extension', () => {
|
||||
expect(handler.canHandle({ filename: 'data.tsv' })).toBe(true)
|
||||
})
|
||||
|
||||
it('should handle .txt extension', () => {
|
||||
expect(handler.canHandle({ filename: 'data.txt' })).toBe(true)
|
||||
})
|
||||
|
||||
it('should handle CSV content (comma)', () => {
|
||||
const content = 'name,age,email\nJohn,30,john@example.com'
|
||||
expect(handler.canHandle(content)).toBe(true)
|
||||
})
|
||||
|
||||
it('should handle CSV content (semicolon)', () => {
|
||||
const content = 'name;age;email\nJohn;30;john@example.com'
|
||||
expect(handler.canHandle(content)).toBe(true)
|
||||
})
|
||||
|
||||
it('should not handle non-CSV content', () => {
|
||||
const content = 'This is just plain text without any structure'
|
||||
expect(handler.canHandle(content)).toBe(false)
|
||||
})
|
||||
|
||||
it('should not handle .xlsx extension', () => {
|
||||
expect(handler.canHandle({ filename: 'data.xlsx' })).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - simple CSV', () => {
|
||||
it('should parse simple comma-delimited CSV', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.csv')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.csv' })
|
||||
|
||||
expect(result.format).toBe('csv')
|
||||
expect(result.data).toHaveLength(4)
|
||||
expect(result.data[0]).toEqual({
|
||||
name: 'John Doe',
|
||||
age: 30,
|
||||
email: 'john@example.com',
|
||||
active: true
|
||||
})
|
||||
expect(result.metadata.rowCount).toBe(4)
|
||||
expect(result.metadata.fields).toEqual(['name', 'age', 'email', 'active'])
|
||||
expect(result.metadata.delimiter).toBe(',')
|
||||
expect(result.metadata.hasHeaders).toBe(true)
|
||||
})
|
||||
|
||||
it('should infer correct types', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.csv')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.csv' })
|
||||
|
||||
expect(typeof result.data[0].name).toBe('string')
|
||||
expect(typeof result.data[0].age).toBe('number')
|
||||
expect(typeof result.data[0].active).toBe('boolean')
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - delimiter detection', () => {
|
||||
it('should detect semicolon delimiter', async () => {
|
||||
const filePath = path.join(fixturesPath, 'semicolon.csv')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'semicolon.csv' })
|
||||
|
||||
expect(result.metadata.delimiter).toBe(';')
|
||||
expect(result.data).toHaveLength(4)
|
||||
expect(result.data[0]).toEqual({
|
||||
product: 'Laptop',
|
||||
price: 999.99,
|
||||
quantity: 10,
|
||||
inStock: true
|
||||
})
|
||||
})
|
||||
|
||||
it('should detect tab delimiter', async () => {
|
||||
const filePath = path.join(fixturesPath, 'tab-delimited.csv')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'tab-delimited.csv' })
|
||||
|
||||
expect(result.metadata.delimiter).toBe('\t')
|
||||
expect(result.data).toHaveLength(4)
|
||||
expect(result.data[0].city).toBe('Tokyo')
|
||||
expect(result.data[0].population).toBe(13960000)
|
||||
})
|
||||
|
||||
it('should use specified delimiter when provided', async () => {
|
||||
const content = 'name|age|email\nJohn|30|john@example.com'
|
||||
|
||||
const result = await handler.process(content, {
|
||||
csvDelimiter: '|',
|
||||
filename: 'pipe.csv'
|
||||
})
|
||||
|
||||
expect(result.data).toHaveLength(1)
|
||||
expect(result.data[0].name).toBe('John')
|
||||
expect(result.data[0].age).toBe(30)
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - type inference', () => {
|
||||
it('should infer integer, float, date, and boolean types', async () => {
|
||||
const filePath = path.join(fixturesPath, 'types.csv')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'types.csv' })
|
||||
|
||||
expect(result.metadata.types).toBeDefined()
|
||||
expect(result.metadata.types.id).toBe('integer')
|
||||
expect(result.metadata.types.name).toBe('string')
|
||||
// Score is 95 (integer), not 95.0, so it's detected as integer
|
||||
expect(['integer', 'float']).toContain(result.metadata.types.score)
|
||||
expect(result.metadata.types.active).toBe('boolean')
|
||||
|
||||
// Check actual values
|
||||
expect(result.data[0].id).toBe(1)
|
||||
expect([95, 95.0]).toContain(result.data[0].score) // Can be integer or float
|
||||
expect(result.data[0].active).toBe(true)
|
||||
expect(result.data[1].score).toBe(87.5) // This one is definitely a float
|
||||
})
|
||||
|
||||
it('should handle null values', async () => {
|
||||
const content = 'name,age,email\nJohn,30,john@example.com\nJane,,jane@example.com\nBob,45,'
|
||||
|
||||
const result = await handler.process(content, { filename: 'nulls.csv' })
|
||||
|
||||
expect(result.data[1].age).toBeNull()
|
||||
expect(result.data[2].email).toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - encoding detection', () => {
|
||||
it('should detect UTF-8 encoding', async () => {
|
||||
const content = 'name,description\nProduct 1,Description with émojis 🎉'
|
||||
|
||||
const result = await handler.process(content, { filename: 'utf8.csv' })
|
||||
|
||||
// Encoding can be 'UTF-8', 'utf-8', or 'utf8' depending on detection
|
||||
expect(result.metadata.encoding.toLowerCase()).toContain('utf')
|
||||
expect(result.data[0].description).toContain('émojis')
|
||||
expect(result.data[0].description).toContain('🎉')
|
||||
})
|
||||
|
||||
it('should use specified encoding when provided', async () => {
|
||||
const content = 'name,age\nJohn,30'
|
||||
|
||||
const result = await handler.process(content, {
|
||||
encoding: 'ascii',
|
||||
filename: 'ascii.csv'
|
||||
})
|
||||
|
||||
expect(result.metadata.encoding).toBe('ascii')
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - options', () => {
|
||||
it('should respect csvHeaders=false option', async () => {
|
||||
const content = 'John,30,john@example.com\nJane,25,jane@example.com'
|
||||
|
||||
const result = await handler.process(content, {
|
||||
csvHeaders: false,
|
||||
filename: 'no-headers.csv'
|
||||
})
|
||||
|
||||
expect(result.metadata.hasHeaders).toBe(false)
|
||||
// csv-parse will generate column names like '0', '1', '2'
|
||||
expect(Object.keys(result.data[0])).toHaveLength(3)
|
||||
})
|
||||
|
||||
it('should respect maxRows option', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.csv')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, {
|
||||
maxRows: 2,
|
||||
filename: 'simple.csv'
|
||||
})
|
||||
|
||||
expect(result.data).toHaveLength(2)
|
||||
expect(result.metadata.rowCount).toBe(2)
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - edge cases', () => {
|
||||
it('should handle empty CSV', async () => {
|
||||
const content = ''
|
||||
|
||||
const result = await handler.process(content, { filename: 'empty.csv' })
|
||||
|
||||
expect(result.data).toHaveLength(0)
|
||||
expect(result.metadata.rowCount).toBe(0)
|
||||
expect(result.metadata.fields).toEqual([])
|
||||
})
|
||||
|
||||
it('should handle CSV with only headers', async () => {
|
||||
const content = 'name,age,email'
|
||||
|
||||
const result = await handler.process(content, { filename: 'headers-only.csv' })
|
||||
|
||||
expect(result.data).toHaveLength(0)
|
||||
expect(result.metadata.rowCount).toBe(0)
|
||||
})
|
||||
|
||||
it('should handle CSV with single row', async () => {
|
||||
const content = 'name,age,email\nJohn,30,john@example.com'
|
||||
|
||||
const result = await handler.process(content, { filename: 'single-row.csv' })
|
||||
|
||||
expect(result.data).toHaveLength(1)
|
||||
expect(result.data[0].name).toBe('John')
|
||||
})
|
||||
|
||||
it('should handle CSV with varying column counts', async () => {
|
||||
const content = 'name,age,email\nJohn,30,john@example.com\nJane,25\nBob,45,bob@example.com,extra'
|
||||
|
||||
const result = await handler.process(content, { filename: 'varying.csv' })
|
||||
|
||||
// csv-parse with relax_column_count should handle this
|
||||
expect(result.data).toHaveLength(3)
|
||||
})
|
||||
|
||||
it('should trim whitespace', async () => {
|
||||
const content = 'name, age , email \n John , 30 , john@example.com '
|
||||
|
||||
const result = await handler.process(content, { filename: 'whitespace.csv' })
|
||||
|
||||
expect(result.data[0].name).toBe('John')
|
||||
expect(result.data[0].age).toBe(30)
|
||||
expect(result.data[0].email).toBe('john@example.com')
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - performance', () => {
|
||||
it('should measure processing time', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.csv')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.csv' })
|
||||
|
||||
expect(result.metadata.processingTime).toBeGreaterThanOrEqual(0)
|
||||
expect(typeof result.metadata.processingTime).toBe('number')
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - error handling', () => {
|
||||
it('should throw error for malformed CSV', async () => {
|
||||
const content = 'name,age,email\nJohn,30,"unclosed quote'
|
||||
|
||||
await expect(
|
||||
handler.process(content, { filename: 'malformed.csv' })
|
||||
).rejects.toThrow('CSV parsing failed')
|
||||
})
|
||||
})
|
||||
})
|
||||
269
tests/augmentations/intelligentImport/excelHandler.unit.test.ts
Normal file
269
tests/augmentations/intelligentImport/excelHandler.unit.test.ts
Normal file
|
|
@ -0,0 +1,269 @@
|
|||
/**
|
||||
* Excel Handler Tests
|
||||
* Comprehensive tests for Excel parsing, multi-sheet extraction, and type inference
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll } from 'vitest'
|
||||
import { ExcelHandler } from '../../../src/augmentations/intelligentImport/handlers/excelHandler.js'
|
||||
import { promises as fs } from 'fs'
|
||||
import * as path from 'path'
|
||||
|
||||
describe('ExcelHandler', () => {
|
||||
let handler: ExcelHandler
|
||||
const fixturesPath = path.join(process.cwd(), 'tests/fixtures/import')
|
||||
|
||||
beforeAll(() => {
|
||||
handler = new ExcelHandler()
|
||||
})
|
||||
|
||||
describe('canHandle', () => {
|
||||
it('should handle .xlsx extension', () => {
|
||||
expect(handler.canHandle({ filename: 'data.xlsx' })).toBe(true)
|
||||
})
|
||||
|
||||
it('should handle .xls extension', () => {
|
||||
expect(handler.canHandle({ filename: 'data.xls' })).toBe(true)
|
||||
})
|
||||
|
||||
it('should handle .xlsb extension', () => {
|
||||
expect(handler.canHandle({ filename: 'data.xlsb' })).toBe(true)
|
||||
})
|
||||
|
||||
it('should handle .xlsm extension', () => {
|
||||
expect(handler.canHandle({ filename: 'data.xlsm' })).toBe(true)
|
||||
})
|
||||
|
||||
it('should not handle .csv extension', () => {
|
||||
expect(handler.canHandle({ filename: 'data.csv' })).toBe(false)
|
||||
})
|
||||
|
||||
it('should not handle .pdf extension', () => {
|
||||
expect(handler.canHandle({ filename: 'data.pdf' })).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - simple Excel', () => {
|
||||
it('should parse simple single-sheet Excel file', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.xlsx')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.xlsx' })
|
||||
|
||||
expect(result.format).toBe('excel')
|
||||
expect(result.data).toHaveLength(4)
|
||||
expect(result.data[0]).toMatchObject({
|
||||
Name: 'Alice Johnson',
|
||||
Age: 28,
|
||||
Department: 'Engineering',
|
||||
Salary: 95000,
|
||||
Active: true
|
||||
})
|
||||
expect(result.metadata.rowCount).toBe(4)
|
||||
expect(result.metadata.sheetCount).toBe(1)
|
||||
expect(result.metadata.sheets).toEqual(['Employees'])
|
||||
})
|
||||
|
||||
it('should infer correct types from Excel data', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.xlsx')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.xlsx' })
|
||||
|
||||
expect(result.metadata.types).toBeDefined()
|
||||
expect(result.metadata.types.Name).toBe('string')
|
||||
expect(result.metadata.types.Age).toBe('integer')
|
||||
expect(['integer', 'float']).toContain(result.metadata.types.Salary)
|
||||
expect(result.metadata.types.Active).toBe('boolean')
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - multi-sheet Excel', () => {
|
||||
it('should extract data from all sheets by default', async () => {
|
||||
const filePath = path.join(fixturesPath, 'multi-sheet.xlsx')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'multi-sheet.xlsx' })
|
||||
|
||||
expect(result.metadata.sheetCount).toBe(3)
|
||||
expect(result.metadata.sheets).toEqual(['Products', 'Orders', 'Customers'])
|
||||
|
||||
// Total rows from all sheets
|
||||
const totalRows = 4 + 3 + 3 // Products + Orders + Customers
|
||||
expect(result.data).toHaveLength(totalRows)
|
||||
|
||||
// Check _sheet field is added
|
||||
expect(result.data[0]._sheet).toBe('Products')
|
||||
})
|
||||
|
||||
it('should filter data by sheet name', async () => {
|
||||
const filePath = path.join(fixturesPath, 'multi-sheet.xlsx')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, {
|
||||
filename: 'multi-sheet.xlsx',
|
||||
excelSheets: ['Products']
|
||||
})
|
||||
|
||||
expect(result.metadata.sheetCount).toBe(1)
|
||||
expect(result.metadata.sheets).toEqual(['Products'])
|
||||
expect(result.data).toHaveLength(4)
|
||||
expect(result.data.every(row => row._sheet === 'Products')).toBe(true)
|
||||
})
|
||||
|
||||
it('should extract data from multiple specified sheets', async () => {
|
||||
const filePath = path.join(fixturesPath, 'multi-sheet.xlsx')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, {
|
||||
filename: 'multi-sheet.xlsx',
|
||||
excelSheets: ['Products', 'Customers']
|
||||
})
|
||||
|
||||
expect(result.metadata.sheetCount).toBe(2)
|
||||
expect(result.metadata.sheets).toEqual(['Products', 'Customers'])
|
||||
expect(result.data).toHaveLength(7) // 4 + 3
|
||||
|
||||
const sheets = new Set(result.data.map(row => row._sheet))
|
||||
expect(sheets.has('Products')).toBe(true)
|
||||
expect(sheets.has('Customers')).toBe(true)
|
||||
expect(sheets.has('Orders')).toBe(false)
|
||||
})
|
||||
|
||||
it('should include sheet metadata for each sheet', async () => {
|
||||
const filePath = path.join(fixturesPath, 'multi-sheet.xlsx')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'multi-sheet.xlsx' })
|
||||
|
||||
expect(result.metadata.sheetMetadata).toBeDefined()
|
||||
expect(result.metadata.sheetMetadata.Products).toMatchObject({
|
||||
rowCount: 4,
|
||||
columnCount: 5
|
||||
})
|
||||
expect(result.metadata.sheetMetadata.Products.headers).toEqual([
|
||||
'ID', 'Product', 'Price', 'Stock', 'Category'
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - type inference', () => {
|
||||
it('should infer integer, float, date, and boolean types', async () => {
|
||||
const filePath = path.join(fixturesPath, 'types.xlsx')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'types.xlsx' })
|
||||
|
||||
expect(result.metadata.types).toBeDefined()
|
||||
expect(result.metadata.types.ID).toBe('integer')
|
||||
expect(result.metadata.types.Name).toBe('string')
|
||||
expect(['integer', 'float']).toContain(result.metadata.types.Score)
|
||||
expect(['integer', 'float']).toContain(result.metadata.types.Percentage)
|
||||
expect(result.metadata.types.Active).toBe('boolean')
|
||||
|
||||
// Check actual values
|
||||
expect(result.data[0].ID).toBe(1)
|
||||
expect(result.data[0].Active).toBe(true)
|
||||
expect(typeof result.data[0].Score).toBe('number')
|
||||
})
|
||||
|
||||
it('should handle null/empty values', async () => {
|
||||
const filePath = path.join(fixturesPath, 'empty.xlsx')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'empty.xlsx' })
|
||||
|
||||
expect(result.data).toHaveLength(0)
|
||||
expect(result.metadata.rowCount).toBe(0)
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - workbook metadata', () => {
|
||||
it('should extract workbook information', async () => {
|
||||
const filePath = path.join(fixturesPath, 'multi-sheet.xlsx')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'multi-sheet.xlsx' })
|
||||
|
||||
expect(result.metadata.workbookInfo).toBeDefined()
|
||||
expect(result.metadata.workbookInfo.sheetNames).toEqual([
|
||||
'Products', 'Orders', 'Customers'
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - edge cases', () => {
|
||||
it('should handle Excel file with only headers', async () => {
|
||||
const filePath = path.join(fixturesPath, 'empty.xlsx')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'empty.xlsx' })
|
||||
|
||||
expect(result.data).toHaveLength(0)
|
||||
expect(result.metadata.sheetCount).toBe(1)
|
||||
})
|
||||
|
||||
it('should skip non-existent sheets when filtering', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.xlsx')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, {
|
||||
filename: 'simple.xlsx',
|
||||
excelSheets: ['Employees', 'NonExistent']
|
||||
})
|
||||
|
||||
expect(result.metadata.sheets).toEqual(['Employees'])
|
||||
expect(result.data).toHaveLength(4)
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - performance', () => {
|
||||
it('should measure processing time', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.xlsx')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.xlsx' })
|
||||
|
||||
expect(result.metadata.processingTime).toBeGreaterThanOrEqual(0)
|
||||
expect(typeof result.metadata.processingTime).toBe('number')
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - field sanitization', () => {
|
||||
it('should sanitize column names', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.xlsx')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.xlsx' })
|
||||
|
||||
// All field names should be valid identifiers
|
||||
const fields = Object.keys(result.data[0]).filter(k => k !== '_sheet')
|
||||
for (const field of fields) {
|
||||
expect(field).toMatch(/^[a-zA-Z_][a-zA-Z0-9_]*$/)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - error handling', () => {
|
||||
it('should handle corrupted Excel file gracefully', async () => {
|
||||
// XLSX library is very forgiving, so we test that it returns empty data
|
||||
// rather than crashing
|
||||
const invalidData = Buffer.from('This is not an Excel file')
|
||||
|
||||
const result = await handler.process(invalidData, { filename: 'invalid.xlsx' })
|
||||
|
||||
// Should return empty data instead of crashing
|
||||
expect(result.data).toHaveLength(0)
|
||||
expect(result.metadata.rowCount).toBe(0)
|
||||
})
|
||||
|
||||
it('should handle binary garbage', async () => {
|
||||
const garbage = Buffer.from(new Uint8Array(256).fill(255))
|
||||
|
||||
const result = await handler.process(garbage, { filename: 'garbage.xlsx' })
|
||||
|
||||
// Should not crash
|
||||
expect(result).toBeDefined()
|
||||
expect(result.format).toBe('excel')
|
||||
})
|
||||
})
|
||||
})
|
||||
292
tests/augmentations/intelligentImport/pdfHandler.unit.test.ts
Normal file
292
tests/augmentations/intelligentImport/pdfHandler.unit.test.ts
Normal file
|
|
@ -0,0 +1,292 @@
|
|||
/**
|
||||
* PDF Handler Tests
|
||||
* Comprehensive tests for PDF text extraction, table detection, and metadata
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll } from 'vitest'
|
||||
import { PDFHandler } from '../../../src/augmentations/intelligentImport/handlers/pdfHandler.js'
|
||||
import { promises as fs } from 'fs'
|
||||
import * as path from 'path'
|
||||
|
||||
describe('PDFHandler', () => {
|
||||
let handler: PDFHandler
|
||||
const fixturesPath = path.join(process.cwd(), 'tests/fixtures/import')
|
||||
|
||||
beforeAll(() => {
|
||||
handler = new PDFHandler()
|
||||
})
|
||||
|
||||
describe('canHandle', () => {
|
||||
it('should handle .pdf extension', () => {
|
||||
expect(handler.canHandle({ filename: 'document.pdf' })).toBe(true)
|
||||
})
|
||||
|
||||
it('should handle PDF magic bytes', () => {
|
||||
const pdfBuffer = Buffer.from('%PDF-1.4\n')
|
||||
expect(handler.canHandle(pdfBuffer)).toBe(true)
|
||||
})
|
||||
|
||||
it('should not handle .xlsx extension', () => {
|
||||
expect(handler.canHandle({ filename: 'data.xlsx' })).toBe(false)
|
||||
})
|
||||
|
||||
it('should not handle .csv extension', () => {
|
||||
expect(handler.canHandle({ filename: 'data.csv' })).toBe(false)
|
||||
})
|
||||
|
||||
it('should not handle non-PDF buffer', () => {
|
||||
const buffer = Buffer.from('This is not a PDF')
|
||||
expect(handler.canHandle(buffer)).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - simple text PDF', () => {
|
||||
it('should extract text from simple PDF', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.pdf' })
|
||||
|
||||
expect(result.format).toBe('pdf')
|
||||
expect(result.data.length).toBeGreaterThan(0)
|
||||
|
||||
// Should have extracted paragraphs
|
||||
const paragraphs = result.data.filter(item => item._type === 'paragraph')
|
||||
expect(paragraphs.length).toBeGreaterThan(0)
|
||||
|
||||
// Check that text was extracted
|
||||
const hasText = paragraphs.some(p => p.text && p.text.length > 0)
|
||||
expect(hasText).toBe(true)
|
||||
})
|
||||
|
||||
it('should include page numbers', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.pdf' })
|
||||
|
||||
// All items should have _page field
|
||||
expect(result.data.every(item => typeof item._page === 'number')).toBe(true)
|
||||
expect(result.data.every(item => item._page >= 1)).toBe(true)
|
||||
})
|
||||
|
||||
it('should count pages correctly', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.pdf' })
|
||||
|
||||
expect(result.metadata.pageCount).toBe(1)
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - multi-page PDF', () => {
|
||||
it('should extract text from all pages', async () => {
|
||||
const filePath = path.join(fixturesPath, 'multi-page.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'multi-page.pdf' })
|
||||
|
||||
expect(result.metadata.pageCount).toBe(3)
|
||||
|
||||
// Should have content from multiple pages
|
||||
const pages = new Set(result.data.map(item => item._page))
|
||||
expect(pages.size).toBeGreaterThan(1)
|
||||
expect(pages.has(1)).toBe(true)
|
||||
expect(pages.has(2)).toBe(true)
|
||||
expect(pages.has(3)).toBe(true)
|
||||
})
|
||||
|
||||
it('should preserve page order', async () => {
|
||||
const filePath = path.join(fixturesPath, 'multi-page.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'multi-page.pdf' })
|
||||
|
||||
// Page numbers should be in order
|
||||
let lastPage = 0
|
||||
let pageChanged = false
|
||||
|
||||
for (const item of result.data) {
|
||||
if (item._page !== lastPage && lastPage > 0) {
|
||||
pageChanged = true
|
||||
}
|
||||
expect(item._page).toBeGreaterThanOrEqual(lastPage)
|
||||
lastPage = item._page
|
||||
}
|
||||
|
||||
expect(pageChanged).toBe(true) // Should have moved through pages
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - table detection', () => {
|
||||
it('should detect tables in PDF', async () => {
|
||||
const filePath = path.join(fixturesPath, 'table.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'table.pdf' })
|
||||
|
||||
// Should have detected at least one table
|
||||
expect(result.metadata.tableCount).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
it('should extract table rows when tables detected', async () => {
|
||||
const filePath = path.join(fixturesPath, 'table.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, {
|
||||
filename: 'table.pdf',
|
||||
pdfExtractTables: true
|
||||
})
|
||||
|
||||
// Should have table_row items
|
||||
const tableRows = result.data.filter(item => item._type === 'table_row')
|
||||
expect(tableRows.length).toBeGreaterThan(0)
|
||||
|
||||
// Table rows should have structured data
|
||||
if (tableRows.length > 0) {
|
||||
const firstRow = tableRows[0]
|
||||
const fields = Object.keys(firstRow).filter(k => !k.startsWith('_'))
|
||||
expect(fields.length).toBeGreaterThan(0)
|
||||
}
|
||||
})
|
||||
|
||||
it('should skip table extraction when disabled', async () => {
|
||||
const filePath = path.join(fixturesPath, 'table.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, {
|
||||
filename: 'table.pdf',
|
||||
pdfExtractTables: false
|
||||
})
|
||||
|
||||
// Should not have detected tables
|
||||
expect(result.metadata.tableCount).toBe(0)
|
||||
|
||||
// Should not have table_row items
|
||||
const tableRows = result.data.filter(item => item._type === 'table_row')
|
||||
expect(tableRows.length).toBe(0)
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - metadata extraction', () => {
|
||||
it('should extract PDF metadata', async () => {
|
||||
const filePath = path.join(fixturesPath, 'metadata.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'metadata.pdf' })
|
||||
|
||||
expect(result.metadata.pdfMetadata).toBeDefined()
|
||||
expect(result.metadata.pdfMetadata.title).toBe('Test Document')
|
||||
expect(result.metadata.pdfMetadata.author).toBe('Test Author')
|
||||
expect(result.metadata.pdfMetadata.creator).toBe('Brainy Test Suite')
|
||||
})
|
||||
|
||||
it('should handle PDFs without metadata gracefully', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.pdf' })
|
||||
|
||||
expect(result.metadata.pdfMetadata).toBeDefined()
|
||||
// Some fields may be null
|
||||
expect(result.metadata.pdfMetadata).toHaveProperty('title')
|
||||
expect(result.metadata.pdfMetadata).toHaveProperty('author')
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - text statistics', () => {
|
||||
it('should track total text length', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.pdf' })
|
||||
|
||||
expect(result.metadata.textLength).toBeGreaterThan(0)
|
||||
expect(typeof result.metadata.textLength).toBe('number')
|
||||
})
|
||||
|
||||
it('should count extracted items', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.pdf' })
|
||||
|
||||
expect(result.metadata.rowCount).toBe(result.data.length)
|
||||
expect(result.metadata.rowCount).toBeGreaterThan(0)
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - edge cases', () => {
|
||||
it('should handle empty PDF', async () => {
|
||||
const filePath = path.join(fixturesPath, 'empty.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'empty.pdf' })
|
||||
|
||||
expect(result.format).toBe('pdf')
|
||||
expect(result.metadata.pageCount).toBe(1)
|
||||
// Empty PDF may have 0 or minimal content
|
||||
expect(result.data).toBeDefined()
|
||||
})
|
||||
|
||||
it('should measure processing time', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.pdf' })
|
||||
|
||||
expect(result.metadata.processingTime).toBeGreaterThan(0)
|
||||
expect(typeof result.metadata.processingTime).toBe('number')
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - data structure', () => {
|
||||
it('should include type indicators', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.pdf' })
|
||||
|
||||
// All items should have _type field
|
||||
expect(result.data.every(item => '_type' in item)).toBe(true)
|
||||
|
||||
// Types should be valid
|
||||
const validTypes = ['paragraph', 'table_row']
|
||||
expect(result.data.every(item => validTypes.includes(item._type))).toBe(true)
|
||||
})
|
||||
|
||||
it('should include index for paragraphs', async () => {
|
||||
const filePath = path.join(fixturesPath, 'simple.pdf')
|
||||
const buffer = await fs.readFile(filePath)
|
||||
|
||||
const result = await handler.process(buffer, { filename: 'simple.pdf' })
|
||||
|
||||
const paragraphs = result.data.filter(item => item._type === 'paragraph')
|
||||
|
||||
// Paragraphs should have _index field
|
||||
expect(paragraphs.every(p => typeof p._index === 'number')).toBe(true)
|
||||
})
|
||||
})
|
||||
|
||||
describe('process - error handling', () => {
|
||||
it('should throw error for invalid PDF', async () => {
|
||||
const invalidData = Buffer.from('This is not a PDF file')
|
||||
|
||||
await expect(
|
||||
handler.process(invalidData, { filename: 'invalid.pdf' })
|
||||
).rejects.toThrow('PDF parsing failed')
|
||||
})
|
||||
|
||||
it('should throw error for corrupted PDF', async () => {
|
||||
const corruptedPDF = Buffer.concat([
|
||||
Buffer.from('%PDF-1.4\n'),
|
||||
Buffer.from('corrupted data that is not valid PDF structure')
|
||||
])
|
||||
|
||||
await expect(
|
||||
handler.process(corruptedPDF, { filename: 'corrupted.pdf' })
|
||||
).rejects.toThrow('PDF parsing failed')
|
||||
})
|
||||
})
|
||||
})
|
||||
233
tests/fixtures/import/empty.pdf
vendored
Normal file
233
tests/fixtures/import/empty.pdf
vendored
Normal file
|
|
@ -0,0 +1,233 @@
|
|||
%PDF-1.3
|
||||
%ºß¬à
|
||||
3 0 obj
|
||||
<</Type /Page
|
||||
/Parent 1 0 R
|
||||
/Resources 2 0 R
|
||||
/MediaBox [0 0 595.2799999999999727 841.8899999999999864]
|
||||
/Contents 4 0 R
|
||||
>>
|
||||
endobj
|
||||
4 0 obj
|
||||
<<
|
||||
/Length 24
|
||||
>>
|
||||
stream
|
||||
0.5670000000000001 w
|
||||
0 G
|
||||
endstream
|
||||
endobj
|
||||
1 0 obj
|
||||
<</Type /Pages
|
||||
/Kids [3 0 R ]
|
||||
/Count 1
|
||||
>>
|
||||
endobj
|
||||
5 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
6 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica-Bold
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
7 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica-Oblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
8 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica-BoldOblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
9 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
10 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier-Bold
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
11 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier-Oblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
12 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier-BoldOblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
13 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-Roman
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
14 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-Bold
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
15 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-Italic
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
16 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-BoldItalic
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
17 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /ZapfDingbats
|
||||
/Subtype /Type1
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
18 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Symbol
|
||||
/Subtype /Type1
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
2 0 obj
|
||||
<<
|
||||
/ProcSet [/PDF /Text /ImageB /ImageC /ImageI]
|
||||
/Font <<
|
||||
/F1 5 0 R
|
||||
/F2 6 0 R
|
||||
/F3 7 0 R
|
||||
/F4 8 0 R
|
||||
/F5 9 0 R
|
||||
/F6 10 0 R
|
||||
/F7 11 0 R
|
||||
/F8 12 0 R
|
||||
/F9 13 0 R
|
||||
/F10 14 0 R
|
||||
/F11 15 0 R
|
||||
/F12 16 0 R
|
||||
/F13 17 0 R
|
||||
/F14 18 0 R
|
||||
>>
|
||||
/XObject <<
|
||||
>>
|
||||
>>
|
||||
endobj
|
||||
19 0 obj
|
||||
<<
|
||||
/Producer (jsPDF 3.0.3)
|
||||
/CreationDate (D:20251001162659-07'00')
|
||||
>>
|
||||
endobj
|
||||
20 0 obj
|
||||
<<
|
||||
/Type /Catalog
|
||||
/Pages 1 0 R
|
||||
/OpenAction [3 0 R /FitH null]
|
||||
/PageLayout /OneColumn
|
||||
>>
|
||||
endobj
|
||||
xref
|
||||
0 21
|
||||
0000000000 65535 f
|
||||
0000000226 00000 n
|
||||
0000002043 00000 n
|
||||
0000000015 00000 n
|
||||
0000000152 00000 n
|
||||
0000000283 00000 n
|
||||
0000000408 00000 n
|
||||
0000000538 00000 n
|
||||
0000000671 00000 n
|
||||
0000000808 00000 n
|
||||
0000000931 00000 n
|
||||
0000001060 00000 n
|
||||
0000001192 00000 n
|
||||
0000001328 00000 n
|
||||
0000001456 00000 n
|
||||
0000001583 00000 n
|
||||
0000001712 00000 n
|
||||
0000001845 00000 n
|
||||
0000001947 00000 n
|
||||
0000002291 00000 n
|
||||
0000002377 00000 n
|
||||
trailer
|
||||
<<
|
||||
/Size 21
|
||||
/Root 20 0 R
|
||||
/Info 19 0 R
|
||||
/ID [ <49861707DD6C8FE0691713F0D6CEE800> <49861707DD6C8FE0691713F0D6CEE800> ]
|
||||
>>
|
||||
startxref
|
||||
2481
|
||||
%%EOF
|
||||
BIN
tests/fixtures/import/empty.xlsx
vendored
Normal file
BIN
tests/fixtures/import/empty.xlsx
vendored
Normal file
Binary file not shown.
109
tests/fixtures/import/generate-excel.ts
vendored
Normal file
109
tests/fixtures/import/generate-excel.ts
vendored
Normal file
|
|
@ -0,0 +1,109 @@
|
|||
/**
|
||||
* Script to generate Excel test fixtures
|
||||
* Run with: npx tsx tests/fixtures/import/generate-excel.ts
|
||||
*/
|
||||
|
||||
import * as XLSX from 'xlsx'
|
||||
import * as path from 'path'
|
||||
import { fileURLToPath } from 'url'
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url)
|
||||
const __dirname = path.dirname(__filename)
|
||||
|
||||
// Simple single-sheet workbook
|
||||
function createSimpleWorkbook() {
|
||||
const data = [
|
||||
['Name', 'Age', 'Department', 'Salary', 'Active'],
|
||||
['Alice Johnson', 28, 'Engineering', 95000, true],
|
||||
['Bob Smith', 35, 'Sales', 75000, true],
|
||||
['Charlie Brown', 42, 'Engineering', 105000, false],
|
||||
['Diana Prince', 31, 'Marketing', 85000, true]
|
||||
]
|
||||
|
||||
const ws = XLSX.utils.aoa_to_sheet(data)
|
||||
const wb = XLSX.utils.book_new()
|
||||
XLSX.utils.book_append_sheet(wb, ws, 'Employees')
|
||||
|
||||
XLSX.writeFile(wb, path.join(__dirname, 'simple.xlsx'))
|
||||
console.log('✅ Created simple.xlsx')
|
||||
}
|
||||
|
||||
// Multi-sheet workbook
|
||||
function createMultiSheetWorkbook() {
|
||||
const wb = XLSX.utils.book_new()
|
||||
|
||||
// Sheet 1: Products
|
||||
const products = [
|
||||
['ID', 'Product', 'Price', 'Stock', 'Category'],
|
||||
[1, 'Laptop', 999.99, 50, 'Electronics'],
|
||||
[2, 'Mouse', 29.99, 200, 'Electronics'],
|
||||
[3, 'Desk', 299.99, 30, 'Furniture'],
|
||||
[4, 'Chair', 199.99, 45, 'Furniture']
|
||||
]
|
||||
const ws1 = XLSX.utils.aoa_to_sheet(products)
|
||||
XLSX.utils.book_append_sheet(wb, ws1, 'Products')
|
||||
|
||||
// Sheet 2: Orders
|
||||
const orders = [
|
||||
['OrderID', 'ProductID', 'Quantity', 'Date', 'Status'],
|
||||
[1001, 1, 2, '2024-01-15', 'Shipped'],
|
||||
[1002, 2, 5, '2024-01-16', 'Delivered'],
|
||||
[1003, 3, 1, '2024-01-17', 'Processing']
|
||||
]
|
||||
const ws2 = XLSX.utils.aoa_to_sheet(orders)
|
||||
XLSX.utils.book_append_sheet(wb, ws2, 'Orders')
|
||||
|
||||
// Sheet 3: Customers
|
||||
const customers = [
|
||||
['CustomerID', 'Name', 'Email', 'Country'],
|
||||
[101, 'John Doe', 'john@example.com', 'USA'],
|
||||
[102, 'Jane Smith', 'jane@example.com', 'Canada'],
|
||||
[103, 'Bob Wilson', 'bob@example.com', 'UK']
|
||||
]
|
||||
const ws3 = XLSX.utils.aoa_to_sheet(customers)
|
||||
XLSX.utils.book_append_sheet(wb, ws3, 'Customers')
|
||||
|
||||
XLSX.writeFile(wb, path.join(__dirname, 'multi-sheet.xlsx'))
|
||||
console.log('✅ Created multi-sheet.xlsx')
|
||||
}
|
||||
|
||||
// Workbook with types
|
||||
function createTypesWorkbook() {
|
||||
const data = [
|
||||
['ID', 'Name', 'Score', 'Percentage', 'Date', 'Active'],
|
||||
[1, 'Alice', 95.5, 0.955, new Date('2024-01-15'), true],
|
||||
[2, 'Bob', 87.3, 0.873, new Date('2024-02-20'), false],
|
||||
[3, 'Charlie', 92.8, 0.928, new Date('2024-03-10'), true]
|
||||
]
|
||||
|
||||
const ws = XLSX.utils.aoa_to_sheet(data)
|
||||
const wb = XLSX.utils.book_new()
|
||||
XLSX.utils.book_append_sheet(wb, ws, 'Scores')
|
||||
|
||||
XLSX.writeFile(wb, path.join(__dirname, 'types.xlsx'))
|
||||
console.log('✅ Created types.xlsx')
|
||||
}
|
||||
|
||||
// Empty workbook (edge case)
|
||||
function createEmptyWorkbook() {
|
||||
const ws = XLSX.utils.aoa_to_sheet([['Header1', 'Header2', 'Header3']])
|
||||
const wb = XLSX.utils.book_new()
|
||||
XLSX.utils.book_append_sheet(wb, ws, 'Empty')
|
||||
|
||||
XLSX.writeFile(wb, path.join(__dirname, 'empty.xlsx'))
|
||||
console.log('✅ Created empty.xlsx')
|
||||
}
|
||||
|
||||
// Generate all fixtures
|
||||
async function main() {
|
||||
console.log('Generating Excel test fixtures...\n')
|
||||
|
||||
createSimpleWorkbook()
|
||||
createMultiSheetWorkbook()
|
||||
createTypesWorkbook()
|
||||
createEmptyWorkbook()
|
||||
|
||||
console.log('\n✨ All Excel fixtures generated!')
|
||||
}
|
||||
|
||||
main().catch(console.error)
|
||||
147
tests/fixtures/import/generate-pdf.ts
vendored
Normal file
147
tests/fixtures/import/generate-pdf.ts
vendored
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
/**
|
||||
* Script to generate PDF test fixtures
|
||||
* Run with: npx tsx tests/fixtures/import/generate-pdf.ts
|
||||
*/
|
||||
|
||||
import { jsPDF } from 'jspdf'
|
||||
import * as path from 'path'
|
||||
import { fileURLToPath } from 'url'
|
||||
import { writeFileSync } from 'fs'
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url)
|
||||
const __dirname = path.dirname(__filename)
|
||||
|
||||
// Simple text PDF
|
||||
function createSimpleTextPDF() {
|
||||
const doc = new jsPDF()
|
||||
|
||||
doc.setFontSize(16)
|
||||
doc.text('Simple PDF Document', 20, 20)
|
||||
|
||||
doc.setFontSize(12)
|
||||
doc.text('This is a test PDF with simple text content.', 20, 40)
|
||||
doc.text('It contains multiple paragraphs to test text extraction.', 20, 50)
|
||||
|
||||
doc.text('Second paragraph starts here.', 20, 70)
|
||||
doc.text('This paragraph has multiple sentences. Each sentence adds more text.', 20, 80)
|
||||
|
||||
doc.text('Third paragraph demonstrates extraction capabilities.', 20, 100)
|
||||
|
||||
const pdfBuffer = Buffer.from(doc.output('arraybuffer'))
|
||||
writeFileSync(path.join(__dirname, 'simple.pdf'), pdfBuffer)
|
||||
console.log('✅ Created simple.pdf')
|
||||
}
|
||||
|
||||
// PDF with table-like structure
|
||||
function createTablePDF() {
|
||||
const doc = new jsPDF()
|
||||
|
||||
doc.setFontSize(14)
|
||||
doc.text('Employee Records', 20, 20)
|
||||
|
||||
// Table headers
|
||||
doc.setFontSize(10)
|
||||
doc.setFont(undefined, 'bold')
|
||||
doc.text('Name', 20, 40)
|
||||
doc.text('Age', 80, 40)
|
||||
doc.text('Department', 120, 40)
|
||||
|
||||
// Table rows
|
||||
doc.setFont(undefined, 'normal')
|
||||
doc.text('Alice Johnson', 20, 50)
|
||||
doc.text('28', 80, 50)
|
||||
doc.text('Engineering', 120, 50)
|
||||
|
||||
doc.text('Bob Smith', 20, 60)
|
||||
doc.text('35', 80, 60)
|
||||
doc.text('Sales', 120, 60)
|
||||
|
||||
doc.text('Charlie Brown', 20, 70)
|
||||
doc.text('42', 80, 70)
|
||||
doc.text('Engineering', 120, 70)
|
||||
|
||||
const pdfBuffer = Buffer.from(doc.output('arraybuffer'))
|
||||
writeFileSync(path.join(__dirname, 'table.pdf'), pdfBuffer)
|
||||
console.log('✅ Created table.pdf')
|
||||
}
|
||||
|
||||
// Multi-page PDF
|
||||
function createMultiPagePDF() {
|
||||
const doc = new jsPDF()
|
||||
|
||||
// Page 1
|
||||
doc.setFontSize(16)
|
||||
doc.text('Page 1: Introduction', 20, 20)
|
||||
doc.setFontSize(12)
|
||||
doc.text('This is the first page of a multi-page PDF document.', 20, 40)
|
||||
doc.text('It demonstrates that the PDF handler can process multiple pages.', 20, 50)
|
||||
|
||||
// Page 2
|
||||
doc.addPage()
|
||||
doc.setFontSize(16)
|
||||
doc.text('Page 2: Content', 20, 20)
|
||||
doc.setFontSize(12)
|
||||
doc.text('This is the second page with different content.', 20, 40)
|
||||
doc.text('Each page should be extracted separately.', 20, 50)
|
||||
|
||||
// Page 3
|
||||
doc.addPage()
|
||||
doc.setFontSize(16)
|
||||
doc.text('Page 3: Conclusion', 20, 20)
|
||||
doc.setFontSize(12)
|
||||
doc.text('This is the final page of the document.', 20, 40)
|
||||
|
||||
const pdfBuffer = Buffer.from(doc.output('arraybuffer'))
|
||||
writeFileSync(path.join(__dirname, 'multi-page.pdf'), pdfBuffer)
|
||||
console.log('✅ Created multi-page.pdf')
|
||||
}
|
||||
|
||||
// PDF with metadata
|
||||
function createMetadataPDF() {
|
||||
const doc = new jsPDF()
|
||||
|
||||
// Set metadata
|
||||
doc.setProperties({
|
||||
title: 'Test Document',
|
||||
subject: 'PDF Testing',
|
||||
author: 'Test Author',
|
||||
keywords: 'pdf, test, metadata',
|
||||
creator: 'Brainy Test Suite'
|
||||
})
|
||||
|
||||
doc.setFontSize(14)
|
||||
doc.text('PDF with Metadata', 20, 20)
|
||||
doc.setFontSize(12)
|
||||
doc.text('This PDF has metadata that should be extracted.', 20, 40)
|
||||
|
||||
const pdfBuffer = Buffer.from(doc.output('arraybuffer'))
|
||||
writeFileSync(path.join(__dirname, 'metadata.pdf'), pdfBuffer)
|
||||
console.log('✅ Created metadata.pdf')
|
||||
}
|
||||
|
||||
// Empty PDF (edge case)
|
||||
function createEmptyPDF() {
|
||||
const doc = new jsPDF()
|
||||
|
||||
// Just a blank page
|
||||
doc.text('', 20, 20)
|
||||
|
||||
const pdfBuffer = Buffer.from(doc.output('arraybuffer'))
|
||||
writeFileSync(path.join(__dirname, 'empty.pdf'), pdfBuffer)
|
||||
console.log('✅ Created empty.pdf')
|
||||
}
|
||||
|
||||
// Generate all fixtures
|
||||
async function main() {
|
||||
console.log('Generating PDF test fixtures...\n')
|
||||
|
||||
createSimpleTextPDF()
|
||||
createTablePDF()
|
||||
createMultiPagePDF()
|
||||
createMetadataPDF()
|
||||
createEmptyPDF()
|
||||
|
||||
console.log('\n✨ All PDF fixtures generated!')
|
||||
}
|
||||
|
||||
main().catch(console.error)
|
||||
252
tests/fixtures/import/metadata.pdf
vendored
Normal file
252
tests/fixtures/import/metadata.pdf
vendored
Normal file
|
|
@ -0,0 +1,252 @@
|
|||
%PDF-1.3
|
||||
%ºß¬à
|
||||
3 0 obj
|
||||
<</Type /Page
|
||||
/Parent 1 0 R
|
||||
/Resources 2 0 R
|
||||
/MediaBox [0 0 595.2799999999999727 841.8899999999999864]
|
||||
/Contents 4 0 R
|
||||
>>
|
||||
endobj
|
||||
4 0 obj
|
||||
<<
|
||||
/Length 274
|
||||
>>
|
||||
stream
|
||||
0.5670000000000001 w
|
||||
0 G
|
||||
BT
|
||||
/F1 14 Tf
|
||||
16.0999999999999979 TL
|
||||
0 g
|
||||
56.6929133858267775 785.1970866141732586 Td
|
||||
(PDF with Metadata) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 12 Tf
|
||||
13.7999999999999989 TL
|
||||
0 g
|
||||
56.6929133858267775 728.5041732283464171 Td
|
||||
(This PDF has metadata that should be extracted.) Tj
|
||||
ET
|
||||
endstream
|
||||
endobj
|
||||
1 0 obj
|
||||
<</Type /Pages
|
||||
/Kids [3 0 R ]
|
||||
/Count 1
|
||||
>>
|
||||
endobj
|
||||
5 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
6 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica-Bold
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
7 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica-Oblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
8 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica-BoldOblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
9 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
10 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier-Bold
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
11 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier-Oblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
12 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier-BoldOblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
13 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-Roman
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
14 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-Bold
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
15 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-Italic
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
16 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-BoldItalic
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
17 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /ZapfDingbats
|
||||
/Subtype /Type1
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
18 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Symbol
|
||||
/Subtype /Type1
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
2 0 obj
|
||||
<<
|
||||
/ProcSet [/PDF /Text /ImageB /ImageC /ImageI]
|
||||
/Font <<
|
||||
/F1 5 0 R
|
||||
/F2 6 0 R
|
||||
/F3 7 0 R
|
||||
/F4 8 0 R
|
||||
/F5 9 0 R
|
||||
/F6 10 0 R
|
||||
/F7 11 0 R
|
||||
/F8 12 0 R
|
||||
/F9 13 0 R
|
||||
/F10 14 0 R
|
||||
/F11 15 0 R
|
||||
/F12 16 0 R
|
||||
/F13 17 0 R
|
||||
/F14 18 0 R
|
||||
>>
|
||||
/XObject <<
|
||||
>>
|
||||
>>
|
||||
endobj
|
||||
19 0 obj
|
||||
<<
|
||||
/Producer (jsPDF 3.0.3)
|
||||
/Title (Test Document)
|
||||
/Subject (PDF Testing)
|
||||
/Author (Test Author)
|
||||
/Keywords (pdf, test, metadata)
|
||||
/Creator (Brainy Test Suite)
|
||||
/CreationDate (D:20251001162659-07'00')
|
||||
>>
|
||||
endobj
|
||||
20 0 obj
|
||||
<<
|
||||
/Type /Catalog
|
||||
/Pages 1 0 R
|
||||
/OpenAction [3 0 R /FitH null]
|
||||
/PageLayout /OneColumn
|
||||
>>
|
||||
endobj
|
||||
xref
|
||||
0 21
|
||||
0000000000 65535 f
|
||||
0000000477 00000 n
|
||||
0000002294 00000 n
|
||||
0000000015 00000 n
|
||||
0000000152 00000 n
|
||||
0000000534 00000 n
|
||||
0000000659 00000 n
|
||||
0000000789 00000 n
|
||||
0000000922 00000 n
|
||||
0000001059 00000 n
|
||||
0000001182 00000 n
|
||||
0000001311 00000 n
|
||||
0000001443 00000 n
|
||||
0000001579 00000 n
|
||||
0000001707 00000 n
|
||||
0000001834 00000 n
|
||||
0000001963 00000 n
|
||||
0000002096 00000 n
|
||||
0000002198 00000 n
|
||||
0000002542 00000 n
|
||||
0000002757 00000 n
|
||||
trailer
|
||||
<<
|
||||
/Size 21
|
||||
/Root 20 0 R
|
||||
/Info 19 0 R
|
||||
/ID [ <A029D212DBC492DA6593A84070F2BDFD> <A029D212DBC492DA6593A84070F2BDFD> ]
|
||||
>>
|
||||
startxref
|
||||
2861
|
||||
%%EOF
|
||||
327
tests/fixtures/import/multi-page.pdf
vendored
Normal file
327
tests/fixtures/import/multi-page.pdf
vendored
Normal file
|
|
@ -0,0 +1,327 @@
|
|||
%PDF-1.3
|
||||
%ºß¬à
|
||||
3 0 obj
|
||||
<</Type /Page
|
||||
/Parent 1 0 R
|
||||
/Resources 2 0 R
|
||||
/MediaBox [0 0 595.2799999999999727 841.8899999999999864]
|
||||
/Contents 4 0 R
|
||||
>>
|
||||
endobj
|
||||
4 0 obj
|
||||
<<
|
||||
/Length 437
|
||||
>>
|
||||
stream
|
||||
0.5670000000000001 w
|
||||
0 G
|
||||
BT
|
||||
/F1 16 Tf
|
||||
18.3999999999999986 TL
|
||||
0 g
|
||||
56.6929133858267775 785.1970866141732586 Td
|
||||
(Page 1: Introduction) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 12 Tf
|
||||
13.7999999999999989 TL
|
||||
0 g
|
||||
56.6929133858267775 728.5041732283464171 Td
|
||||
(This is the first page of a multi-page PDF document.) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 12 Tf
|
||||
13.7999999999999989 TL
|
||||
0 g
|
||||
56.6929133858267775 700.15771653543311 Td
|
||||
(It demonstrates that the PDF handler can process multiple pages.) Tj
|
||||
ET
|
||||
endstream
|
||||
endobj
|
||||
5 0 obj
|
||||
<</Type /Page
|
||||
/Parent 1 0 R
|
||||
/Resources 2 0 R
|
||||
/MediaBox [0 0 595.2799999999999727 841.8899999999999864]
|
||||
/Contents 6 0 R
|
||||
>>
|
||||
endobj
|
||||
6 0 obj
|
||||
<<
|
||||
/Length 404
|
||||
>>
|
||||
stream
|
||||
0.5670000000000001 w
|
||||
0 G
|
||||
BT
|
||||
/F1 16 Tf
|
||||
18.3999999999999986 TL
|
||||
0 g
|
||||
56.6929133858267775 785.1970866141732586 Td
|
||||
(Page 2: Content) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 12 Tf
|
||||
13.7999999999999989 TL
|
||||
0 g
|
||||
56.6929133858267775 728.5041732283464171 Td
|
||||
(This is the second page with different content.) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 12 Tf
|
||||
13.7999999999999989 TL
|
||||
0 g
|
||||
56.6929133858267775 700.15771653543311 Td
|
||||
(Each page should be extracted separately.) Tj
|
||||
ET
|
||||
endstream
|
||||
endobj
|
||||
7 0 obj
|
||||
<</Type /Page
|
||||
/Parent 1 0 R
|
||||
/Resources 2 0 R
|
||||
/MediaBox [0 0 595.2799999999999727 841.8899999999999864]
|
||||
/Contents 8 0 R
|
||||
>>
|
||||
endobj
|
||||
8 0 obj
|
||||
<<
|
||||
/Length 267
|
||||
>>
|
||||
stream
|
||||
0.5670000000000001 w
|
||||
0 G
|
||||
BT
|
||||
/F1 16 Tf
|
||||
18.3999999999999986 TL
|
||||
0 g
|
||||
56.6929133858267775 785.1970866141732586 Td
|
||||
(Page 3: Conclusion) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 12 Tf
|
||||
13.7999999999999989 TL
|
||||
0 g
|
||||
56.6929133858267775 728.5041732283464171 Td
|
||||
(This is the final page of the document.) Tj
|
||||
ET
|
||||
endstream
|
||||
endobj
|
||||
1 0 obj
|
||||
<</Type /Pages
|
||||
/Kids [3 0 R 5 0 R 7 0 R ]
|
||||
/Count 3
|
||||
>>
|
||||
endobj
|
||||
9 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
10 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica-Bold
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
11 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica-Oblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
12 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica-BoldOblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
13 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
14 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier-Bold
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
15 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier-Oblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
16 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier-BoldOblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
17 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-Roman
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
18 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-Bold
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
19 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-Italic
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
20 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-BoldItalic
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
21 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /ZapfDingbats
|
||||
/Subtype /Type1
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
22 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Symbol
|
||||
/Subtype /Type1
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
2 0 obj
|
||||
<<
|
||||
/ProcSet [/PDF /Text /ImageB /ImageC /ImageI]
|
||||
/Font <<
|
||||
/F1 9 0 R
|
||||
/F2 10 0 R
|
||||
/F3 11 0 R
|
||||
/F4 12 0 R
|
||||
/F5 13 0 R
|
||||
/F6 14 0 R
|
||||
/F7 15 0 R
|
||||
/F8 16 0 R
|
||||
/F9 17 0 R
|
||||
/F10 18 0 R
|
||||
/F11 19 0 R
|
||||
/F12 20 0 R
|
||||
/F13 21 0 R
|
||||
/F14 22 0 R
|
||||
>>
|
||||
/XObject <<
|
||||
>>
|
||||
>>
|
||||
endobj
|
||||
23 0 obj
|
||||
<<
|
||||
/Producer (jsPDF 3.0.3)
|
||||
/CreationDate (D:20251001162659-07'00')
|
||||
>>
|
||||
endobj
|
||||
24 0 obj
|
||||
<<
|
||||
/Type /Catalog
|
||||
/Pages 1 0 R
|
||||
/OpenAction [3 0 R /FitH null]
|
||||
/PageLayout /OneColumn
|
||||
>>
|
||||
endobj
|
||||
xref
|
||||
0 25
|
||||
0000000000 65535 f
|
||||
0000001687 00000 n
|
||||
0000003520 00000 n
|
||||
0000000015 00000 n
|
||||
0000000152 00000 n
|
||||
0000000640 00000 n
|
||||
0000000777 00000 n
|
||||
0000001232 00000 n
|
||||
0000001369 00000 n
|
||||
0000001756 00000 n
|
||||
0000001881 00000 n
|
||||
0000002012 00000 n
|
||||
0000002146 00000 n
|
||||
0000002284 00000 n
|
||||
0000002408 00000 n
|
||||
0000002537 00000 n
|
||||
0000002669 00000 n
|
||||
0000002805 00000 n
|
||||
0000002933 00000 n
|
||||
0000003060 00000 n
|
||||
0000003189 00000 n
|
||||
0000003322 00000 n
|
||||
0000003424 00000 n
|
||||
0000003772 00000 n
|
||||
0000003858 00000 n
|
||||
trailer
|
||||
<<
|
||||
/Size 25
|
||||
/Root 24 0 R
|
||||
/Info 23 0 R
|
||||
/ID [ <04E6AF3F674215986B311755247B62DD> <04E6AF3F674215986B311755247B62DD> ]
|
||||
>>
|
||||
startxref
|
||||
3962
|
||||
%%EOF
|
||||
BIN
tests/fixtures/import/multi-sheet.xlsx
vendored
Normal file
BIN
tests/fixtures/import/multi-sheet.xlsx
vendored
Normal file
Binary file not shown.
5
tests/fixtures/import/semicolon.csv
vendored
Normal file
5
tests/fixtures/import/semicolon.csv
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
product;price;quantity;inStock
|
||||
Laptop;999.99;10;yes
|
||||
Mouse;29.99;100;yes
|
||||
Keyboard;79.99;50;no
|
||||
Monitor;299.99;25;yes
|
||||
|
5
tests/fixtures/import/simple.csv
vendored
Normal file
5
tests/fixtures/import/simple.csv
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
name,age,email,active
|
||||
John Doe,30,john@example.com,true
|
||||
Jane Smith,25,jane@example.com,false
|
||||
Bob Johnson,45,bob@example.com,true
|
||||
Alice Williams,35,alice@example.com,true
|
||||
|
275
tests/fixtures/import/simple.pdf
vendored
Normal file
275
tests/fixtures/import/simple.pdf
vendored
Normal file
|
|
@ -0,0 +1,275 @@
|
|||
%PDF-1.3
|
||||
%ºß¬à
|
||||
3 0 obj
|
||||
<</Type /Page
|
||||
/Parent 1 0 R
|
||||
/Resources 2 0 R
|
||||
/MediaBox [0 0 595.2799999999999727 841.8899999999999864]
|
||||
/Contents 4 0 R
|
||||
>>
|
||||
endobj
|
||||
4 0 obj
|
||||
<<
|
||||
/Length 847
|
||||
>>
|
||||
stream
|
||||
0.5670000000000001 w
|
||||
0 G
|
||||
BT
|
||||
/F1 16 Tf
|
||||
18.3999999999999986 TL
|
||||
0 g
|
||||
56.6929133858267775 785.1970866141732586 Td
|
||||
(Simple PDF Document) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 12 Tf
|
||||
13.7999999999999989 TL
|
||||
0 g
|
||||
56.6929133858267775 728.5041732283464171 Td
|
||||
(This is a test PDF with simple text content.) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 12 Tf
|
||||
13.7999999999999989 TL
|
||||
0 g
|
||||
56.6929133858267775 700.15771653543311 Td
|
||||
(It contains multiple paragraphs to test text extraction.) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 12 Tf
|
||||
13.7999999999999989 TL
|
||||
0 g
|
||||
56.6929133858267775 643.4648031496062686 Td
|
||||
(Second paragraph starts here.) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 12 Tf
|
||||
13.7999999999999989 TL
|
||||
0 g
|
||||
56.6929133858267775 615.1183464566928478 Td
|
||||
(This paragraph has multiple sentences. Each sentence adds more text.) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 12 Tf
|
||||
13.7999999999999989 TL
|
||||
0 g
|
||||
56.6929133858267775 558.42543307086612 Td
|
||||
(Third paragraph demonstrates extraction capabilities.) Tj
|
||||
ET
|
||||
endstream
|
||||
endobj
|
||||
1 0 obj
|
||||
<</Type /Pages
|
||||
/Kids [3 0 R ]
|
||||
/Count 1
|
||||
>>
|
||||
endobj
|
||||
5 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
6 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica-Bold
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
7 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica-Oblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
8 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica-BoldOblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
9 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
10 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier-Bold
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
11 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier-Oblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
12 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier-BoldOblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
13 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-Roman
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
14 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-Bold
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
15 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-Italic
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
16 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-BoldItalic
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
17 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /ZapfDingbats
|
||||
/Subtype /Type1
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
18 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Symbol
|
||||
/Subtype /Type1
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
2 0 obj
|
||||
<<
|
||||
/ProcSet [/PDF /Text /ImageB /ImageC /ImageI]
|
||||
/Font <<
|
||||
/F1 5 0 R
|
||||
/F2 6 0 R
|
||||
/F3 7 0 R
|
||||
/F4 8 0 R
|
||||
/F5 9 0 R
|
||||
/F6 10 0 R
|
||||
/F7 11 0 R
|
||||
/F8 12 0 R
|
||||
/F9 13 0 R
|
||||
/F10 14 0 R
|
||||
/F11 15 0 R
|
||||
/F12 16 0 R
|
||||
/F13 17 0 R
|
||||
/F14 18 0 R
|
||||
>>
|
||||
/XObject <<
|
||||
>>
|
||||
>>
|
||||
endobj
|
||||
19 0 obj
|
||||
<<
|
||||
/Producer (jsPDF 3.0.3)
|
||||
/CreationDate (D:20251001162659-07'00')
|
||||
>>
|
||||
endobj
|
||||
20 0 obj
|
||||
<<
|
||||
/Type /Catalog
|
||||
/Pages 1 0 R
|
||||
/OpenAction [3 0 R /FitH null]
|
||||
/PageLayout /OneColumn
|
||||
>>
|
||||
endobj
|
||||
xref
|
||||
0 21
|
||||
0000000000 65535 f
|
||||
0000001050 00000 n
|
||||
0000002867 00000 n
|
||||
0000000015 00000 n
|
||||
0000000152 00000 n
|
||||
0000001107 00000 n
|
||||
0000001232 00000 n
|
||||
0000001362 00000 n
|
||||
0000001495 00000 n
|
||||
0000001632 00000 n
|
||||
0000001755 00000 n
|
||||
0000001884 00000 n
|
||||
0000002016 00000 n
|
||||
0000002152 00000 n
|
||||
0000002280 00000 n
|
||||
0000002407 00000 n
|
||||
0000002536 00000 n
|
||||
0000002669 00000 n
|
||||
0000002771 00000 n
|
||||
0000003115 00000 n
|
||||
0000003201 00000 n
|
||||
trailer
|
||||
<<
|
||||
/Size 21
|
||||
/Root 20 0 R
|
||||
/Info 19 0 R
|
||||
/ID [ <0F5A6C1D434F4F388C03021EE84E0E57> <0F5A6C1D434F4F388C03021EE84E0E57> ]
|
||||
>>
|
||||
startxref
|
||||
3305
|
||||
%%EOF
|
||||
BIN
tests/fixtures/import/simple.xlsx
vendored
Normal file
BIN
tests/fixtures/import/simple.xlsx
vendored
Normal file
Binary file not shown.
5
tests/fixtures/import/tab-delimited.csv
vendored
Normal file
5
tests/fixtures/import/tab-delimited.csv
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
city country population area
|
||||
Tokyo Japan 13960000 2191
|
||||
New York USA 8336000 783.8
|
||||
London UK 9002000 1572
|
||||
Paris France 2161000 105.4
|
||||
|
324
tests/fixtures/import/table.pdf
vendored
Normal file
324
tests/fixtures/import/table.pdf
vendored
Normal file
|
|
@ -0,0 +1,324 @@
|
|||
%PDF-1.3
|
||||
%ºß¬à
|
||||
3 0 obj
|
||||
<</Type /Page
|
||||
/Parent 1 0 R
|
||||
/Resources 2 0 R
|
||||
/MediaBox [0 0 595.2799999999999727 841.8899999999999864]
|
||||
/Contents 4 0 R
|
||||
>>
|
||||
endobj
|
||||
4 0 obj
|
||||
<<
|
||||
/Length 1152
|
||||
>>
|
||||
stream
|
||||
0.5670000000000001 w
|
||||
0 G
|
||||
BT
|
||||
/F1 14 Tf
|
||||
16.0999999999999979 TL
|
||||
0 g
|
||||
56.6929133858267775 785.1970866141732586 Td
|
||||
(Employee Records) Tj
|
||||
ET
|
||||
BT
|
||||
/F2 10 Tf
|
||||
11.5 TL
|
||||
0 g
|
||||
56.6929133858267775 728.5041732283464171 Td
|
||||
(Name) Tj
|
||||
ET
|
||||
BT
|
||||
/F2 10 Tf
|
||||
11.5 TL
|
||||
0 g
|
||||
226.7716535433071101 728.5041732283464171 Td
|
||||
(Age) Tj
|
||||
ET
|
||||
BT
|
||||
/F2 10 Tf
|
||||
11.5 TL
|
||||
0 g
|
||||
340.157480314960651 728.5041732283464171 Td
|
||||
(Department) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 10 Tf
|
||||
11.5 TL
|
||||
0 g
|
||||
56.6929133858267775 700.15771653543311 Td
|
||||
(Alice Johnson) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 10 Tf
|
||||
11.5 TL
|
||||
0 g
|
||||
226.7716535433071101 700.15771653543311 Td
|
||||
(28) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 10 Tf
|
||||
11.5 TL
|
||||
0 g
|
||||
340.157480314960651 700.15771653543311 Td
|
||||
(Engineering) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 10 Tf
|
||||
11.5 TL
|
||||
0 g
|
||||
56.6929133858267775 671.8112598425196893 Td
|
||||
(Bob Smith) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 10 Tf
|
||||
11.5 TL
|
||||
0 g
|
||||
226.7716535433071101 671.8112598425196893 Td
|
||||
(35) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 10 Tf
|
||||
11.5 TL
|
||||
0 g
|
||||
340.157480314960651 671.8112598425196893 Td
|
||||
(Sales) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 10 Tf
|
||||
11.5 TL
|
||||
0 g
|
||||
56.6929133858267775 643.4648031496062686 Td
|
||||
(Charlie Brown) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 10 Tf
|
||||
11.5 TL
|
||||
0 g
|
||||
226.7716535433071101 643.4648031496062686 Td
|
||||
(42) Tj
|
||||
ET
|
||||
BT
|
||||
/F1 10 Tf
|
||||
11.5 TL
|
||||
0 g
|
||||
340.157480314960651 643.4648031496062686 Td
|
||||
(Engineering) Tj
|
||||
ET
|
||||
endstream
|
||||
endobj
|
||||
1 0 obj
|
||||
<</Type /Pages
|
||||
/Kids [3 0 R ]
|
||||
/Count 1
|
||||
>>
|
||||
endobj
|
||||
5 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
6 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica-Bold
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
7 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica-Oblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
8 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Helvetica-BoldOblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
9 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
10 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier-Bold
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
11 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier-Oblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
12 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Courier-BoldOblique
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
13 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-Roman
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
14 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-Bold
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
15 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-Italic
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
16 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Times-BoldItalic
|
||||
/Subtype /Type1
|
||||
/Encoding /WinAnsiEncoding
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
17 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /ZapfDingbats
|
||||
/Subtype /Type1
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
18 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/BaseFont /Symbol
|
||||
/Subtype /Type1
|
||||
/FirstChar 32
|
||||
/LastChar 255
|
||||
>>
|
||||
endobj
|
||||
2 0 obj
|
||||
<<
|
||||
/ProcSet [/PDF /Text /ImageB /ImageC /ImageI]
|
||||
/Font <<
|
||||
/F1 5 0 R
|
||||
/F2 6 0 R
|
||||
/F3 7 0 R
|
||||
/F4 8 0 R
|
||||
/F5 9 0 R
|
||||
/F6 10 0 R
|
||||
/F7 11 0 R
|
||||
/F8 12 0 R
|
||||
/F9 13 0 R
|
||||
/F10 14 0 R
|
||||
/F11 15 0 R
|
||||
/F12 16 0 R
|
||||
/F13 17 0 R
|
||||
/F14 18 0 R
|
||||
>>
|
||||
/XObject <<
|
||||
>>
|
||||
>>
|
||||
endobj
|
||||
19 0 obj
|
||||
<<
|
||||
/Producer (jsPDF 3.0.3)
|
||||
/CreationDate (D:20251001162659-07'00')
|
||||
>>
|
||||
endobj
|
||||
20 0 obj
|
||||
<<
|
||||
/Type /Catalog
|
||||
/Pages 1 0 R
|
||||
/OpenAction [3 0 R /FitH null]
|
||||
/PageLayout /OneColumn
|
||||
>>
|
||||
endobj
|
||||
xref
|
||||
0 21
|
||||
0000000000 65535 f
|
||||
0000001356 00000 n
|
||||
0000003173 00000 n
|
||||
0000000015 00000 n
|
||||
0000000152 00000 n
|
||||
0000001413 00000 n
|
||||
0000001538 00000 n
|
||||
0000001668 00000 n
|
||||
0000001801 00000 n
|
||||
0000001938 00000 n
|
||||
0000002061 00000 n
|
||||
0000002190 00000 n
|
||||
0000002322 00000 n
|
||||
0000002458 00000 n
|
||||
0000002586 00000 n
|
||||
0000002713 00000 n
|
||||
0000002842 00000 n
|
||||
0000002975 00000 n
|
||||
0000003077 00000 n
|
||||
0000003421 00000 n
|
||||
0000003507 00000 n
|
||||
trailer
|
||||
<<
|
||||
/Size 21
|
||||
/Root 20 0 R
|
||||
/Info 19 0 R
|
||||
/ID [ <A34A2614AA2553099D2A387E65B754AA> <A34A2614AA2553099D2A387E65B754AA> ]
|
||||
>>
|
||||
startxref
|
||||
3611
|
||||
%%EOF
|
||||
4
tests/fixtures/import/types.csv
vendored
Normal file
4
tests/fixtures/import/types.csv
vendored
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
id,name,score,percentage,date,timestamp,active
|
||||
1,Alice,95,0.95,2024-01-15,2024-01-15T10:30:00Z,true
|
||||
2,Bob,87.5,0.875,2024-02-20,2024-02-20T14:45:00Z,false
|
||||
3,Charlie,92,0.92,2024-03-10,2024-03-10T09:15:00Z,true
|
||||
|
BIN
tests/fixtures/import/types.xlsx
vendored
Normal file
BIN
tests/fixtures/import/types.xlsx
vendored
Normal file
Binary file not shown.
Loading…
Add table
Add a link
Reference in a new issue