David Snelling
ccd6c541ad
Cleaned up public documentation to remove references to external projects (Workshop). Documentation should be project-agnostic and professional. Changes: - docs/guides/import-progress-examples.md: Changed "Workshop team problem SOLVED" to "Problem SOLVED" - docs/guides/standard-import-progress.md: Changed "Workshop team (and any developer)" to "Developers" 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
370 lines
9.5 KiB
Markdown
370 lines
9.5 KiB
Markdown
# Import Progress - Usage Examples
|
|
|
|
**How to Use Progress Tracking in Your Applications**
|
|
|
|
Brainy v4.5.0+ provides real-time progress tracking for **all 7 supported file formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX).
|
|
|
|
> **⚠️ KEY FEATURE:** The progress API is **100% standardized**. Write your progress handler ONCE and it works for ALL formats with zero format-specific code! See [Standard Import Progress API](./standard-import-progress.md) for the complete interface documentation.
|
|
|
|
---
|
|
|
|
## 🚀 Quick Start
|
|
|
|
### Basic Progress Tracking
|
|
|
|
```typescript
|
|
import { Brainy } from '@soulcraft/brainy'
|
|
import * as fs from 'fs'
|
|
|
|
const brain = await Brainy.create()
|
|
|
|
// Import with progress tracking
|
|
const result = await brain.import(fs.readFileSync('large-file.xlsx'), {
|
|
onProgress: (progress) => {
|
|
console.log(`Progress: ${progress.stage}`)
|
|
console.log(` Message: ${progress.message}`)
|
|
console.log(` Entities: ${progress.entities || 0}`)
|
|
console.log(` Relationships: ${progress.relationships || 0}`)
|
|
}
|
|
})
|
|
|
|
console.log(`Import complete: ${result.entities.length} entities created`)
|
|
```
|
|
|
|
**Expected Output:**
|
|
```
|
|
Progress: detecting
|
|
Message: Detecting format...
|
|
Entities: 0
|
|
Relationships: 0
|
|
Progress: extracting
|
|
Message: Loading Excel workbook...
|
|
Entities: 0
|
|
Relationships: 0
|
|
Progress: extracting
|
|
Message: Reading sheet: Sales (1/3)
|
|
Entities: 0
|
|
Relationships: 0
|
|
Progress: extracting
|
|
Message: Parsing Excel (33%)
|
|
Entities: 0
|
|
Relationships: 0
|
|
Progress: extracting
|
|
Message: Reading sheet: Products (2/3)
|
|
Entities: 0
|
|
Relationships: 0
|
|
... (more progress updates)
|
|
Progress: complete
|
|
Message: Import complete
|
|
Entities: 1523
|
|
Relationships: 892
|
|
Import complete: 1523 entities created
|
|
```
|
|
|
|
---
|
|
|
|
## 🎯 Universal Progress Handler (Works for ALL Formats)
|
|
|
|
The examples below show format-specific messages, but **you don't need format-specific code**! The `ImportProgress` interface is the same for all formats:
|
|
|
|
```typescript
|
|
// ONE HANDLER FOR ALL FORMATS!
|
|
function universalProgressHandler(progress) {
|
|
console.log(`[${progress.stage}] ${progress.message}`)
|
|
|
|
if (progress.processed && progress.total) {
|
|
console.log(` Progress: ${progress.processed}/${progress.total}`)
|
|
}
|
|
|
|
if (progress.entities || progress.relationships) {
|
|
console.log(` Extracted: ${progress.entities || 0} entities, ${progress.relationships || 0} relationships`)
|
|
}
|
|
|
|
if (progress.throughput && progress.eta) {
|
|
console.log(` Rate: ${progress.throughput.toFixed(1)}/sec, ETA: ${Math.round(progress.eta/1000)}s`)
|
|
}
|
|
}
|
|
|
|
// Use it for ANY format!
|
|
await brain.import(csvBuffer, { onProgress: universalProgressHandler })
|
|
await brain.import(pdfBuffer, { onProgress: universalProgressHandler })
|
|
await brain.import(excelBuffer, { onProgress: universalProgressHandler })
|
|
await brain.import(jsonBuffer, { onProgress: universalProgressHandler })
|
|
await brain.import(markdownString, { onProgress: universalProgressHandler })
|
|
await brain.import(yamlBuffer, { onProgress: universalProgressHandler })
|
|
await brain.import(docxBuffer, { onProgress: universalProgressHandler })
|
|
```
|
|
|
|
---
|
|
|
|
## 📊 What Different Formats Look Like (Same Handler!)
|
|
|
|
The examples below show **what messages look like** for different formats using the **same universal handler** above.
|
|
|
|
### CSV Import (Row-by-Row Progress)
|
|
|
|
```typescript
|
|
await brain.import(csvBuffer, {
|
|
format: 'csv',
|
|
onProgress: (progress) => {
|
|
if (progress.stage === 'extracting') {
|
|
// CSV reports: "Parsing CSV (45%)", "Extracted 1000 rows", etc.
|
|
console.log(progress.message)
|
|
}
|
|
}
|
|
})
|
|
```
|
|
|
|
**CSV Progress Messages:**
|
|
- ✅ "Detecting CSV encoding and delimiter..."
|
|
- ✅ "Parsing CSV rows (delimiter: ",")"
|
|
- ✅ "Parsed 75%" (via bytes processed)
|
|
- ✅ "Extracted 1000 rows"
|
|
- ✅ "Converting types: 5000/10000 rows..."
|
|
- ✅ "CSV processing complete: 10000 rows"
|
|
|
|
---
|
|
|
|
### PDF Import (Page-by-Page Progress)
|
|
|
|
```typescript
|
|
await brain.import(pdfBuffer, {
|
|
format: 'pdf',
|
|
onProgress: (progress) => {
|
|
// PDF reports exact page numbers
|
|
console.log(progress.message)
|
|
// Example: "Processing page 5 of 23"
|
|
}
|
|
})
|
|
```
|
|
|
|
**PDF Progress Messages:**
|
|
- ✅ "Loading PDF document..."
|
|
- ✅ "Processing 23 pages..."
|
|
- ✅ "Processing page 5 of 23"
|
|
- ✅ "Parsed 22%" (via bytes processed)
|
|
- ✅ "Extracted 156 items from PDF"
|
|
- ✅ "PDF complete: 23 pages, 156 items extracted"
|
|
|
|
---
|
|
|
|
### Excel Import (Sheet-by-Sheet Progress)
|
|
|
|
```typescript
|
|
await brain.import(excelBuffer, {
|
|
format: 'excel',
|
|
onProgress: (progress) => {
|
|
// Excel reports sheet names
|
|
console.log(progress.message)
|
|
// Example: "Reading sheet: Q2 Sales (2/5)"
|
|
}
|
|
})
|
|
```
|
|
|
|
**Excel Progress Messages:**
|
|
- ✅ "Loading Excel workbook..."
|
|
- ✅ "Processing 3 sheets..."
|
|
- ✅ "Reading sheet: Sales (1/3)"
|
|
- ✅ "Parsing Excel (33%)" (via bytes processed)
|
|
- ✅ "Extracted 5234 rows from Excel"
|
|
- ✅ "Excel complete: 3 sheets, 5234 rows"
|
|
|
|
---
|
|
|
|
### JSON Import (Node Traversal)
|
|
|
|
```typescript
|
|
await brain.import(jsonBuffer, {
|
|
format: 'json',
|
|
onProgress: (progress) => {
|
|
// JSON reports every 10 nodes
|
|
console.log(`Processed ${progress.processed} nodes, found ${progress.entities} entities`)
|
|
}
|
|
})
|
|
```
|
|
|
|
---
|
|
|
|
### Markdown Import (Section-by-Section)
|
|
|
|
```typescript
|
|
await brain.import(markdownString, {
|
|
format: 'markdown',
|
|
onProgress: (progress) => {
|
|
console.log(`Section ${progress.processed}/${progress.total}`)
|
|
}
|
|
})
|
|
```
|
|
|
|
---
|
|
|
|
## 🎯 Building Progress UI Components
|
|
|
|
### React Progress Bar
|
|
|
|
```typescript
|
|
function ImportProgress({ file }: { file: File }) {
|
|
const [progress, setProgress] = useState({
|
|
stage: 'idle',
|
|
message: '',
|
|
percent: 0,
|
|
entities: 0,
|
|
relationships: 0
|
|
})
|
|
|
|
const handleImport = async () => {
|
|
const buffer = await file.arrayBuffer()
|
|
|
|
await brain.import(Buffer.from(buffer), {
|
|
onProgress: (p) => {
|
|
setProgress({
|
|
stage: p.stage,
|
|
message: p.message,
|
|
// Estimate percentage from stage
|
|
percent: {
|
|
detecting: 10,
|
|
extracting: 50,
|
|
'storing-vfs': 80,
|
|
'storing-graph': 90,
|
|
complete: 100
|
|
}[p.stage] || 0,
|
|
entities: p.entities || 0,
|
|
relationships: p.relationships || 0
|
|
})
|
|
}
|
|
})
|
|
}
|
|
|
|
return (
|
|
<div>
|
|
<ProgressBar value={progress.percent} />
|
|
<p>{progress.message}</p>
|
|
<p>Entities: {progress.entities} | Relationships: {progress.relationships}</p>
|
|
</div>
|
|
)
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### CLI Progress Spinner
|
|
|
|
```typescript
|
|
import ora from 'ora'
|
|
|
|
const spinner = ora('Starting import...').start()
|
|
|
|
await brain.import(buffer, {
|
|
onProgress: (progress) => {
|
|
spinner.text = progress.message
|
|
|
|
if (progress.stage === 'complete') {
|
|
spinner.succeed(`Import complete: ${progress.entities} entities`)
|
|
}
|
|
}
|
|
})
|
|
```
|
|
|
|
**CLI Output:**
|
|
```
|
|
⠋ Detecting format...
|
|
⠙ Loading Excel workbook...
|
|
⠹ Reading sheet: Sales (1/3)
|
|
⠸ Parsing Excel (33%)
|
|
⠼ Reading sheet: Products (2/3)
|
|
...
|
|
✔ Import complete: 1523 entities
|
|
```
|
|
|
|
---
|
|
|
|
### Progress Dashboard with ETA
|
|
|
|
```typescript
|
|
let startTime = Date.now()
|
|
let lastUpdate = startTime
|
|
|
|
await brain.import(buffer, {
|
|
onProgress: (progress) => {
|
|
const elapsed = Date.now() - startTime
|
|
const rate = progress.entities / (elapsed / 1000) // entities/sec
|
|
|
|
console.clear()
|
|
console.log('Import Progress Dashboard')
|
|
console.log('========================')
|
|
console.log(`Stage: ${progress.stage}`)
|
|
console.log(`Status: ${progress.message}`)
|
|
console.log(`Entities: ${progress.entities}`)
|
|
console.log(`Relationships: ${progress.relationships}`)
|
|
console.log(`Rate: ${rate.toFixed(1)} entities/sec`)
|
|
console.log(`Elapsed: ${(elapsed / 1000).toFixed(1)}s`)
|
|
}
|
|
})
|
|
```
|
|
|
|
---
|
|
|
|
## 🔧 Advanced: Format-Specific Optimization
|
|
|
|
### Detecting Format to Show Appropriate Progress
|
|
|
|
```typescript
|
|
const formatMessages = {
|
|
csv: (p) => `CSV: ${p.message}`,
|
|
pdf: (p) => `PDF: ${p.message}`,
|
|
excel: (p) => `Excel: ${p.message}`,
|
|
json: (p) => `JSON: ${p.processed} nodes, ${p.entities} entities`,
|
|
markdown: (p) => `Markdown: Section ${p.processed}/${p.total}`,
|
|
yaml: (p) => `YAML: ${p.processed} nodes`,
|
|
docx: (p) => `DOCX: ${p.processed} paragraphs`
|
|
}
|
|
|
|
await brain.import(buffer, {
|
|
onProgress: (progress) => {
|
|
// Format is available in progress.stage metadata
|
|
const message = formatMessages[detectedFormat]?.(progress) || progress.message
|
|
console.log(message)
|
|
}
|
|
})
|
|
```
|
|
|
|
---
|
|
|
|
## ⚡ Performance Tips
|
|
|
|
### Throttle UI Updates
|
|
|
|
```typescript
|
|
let lastUIUpdate = 0
|
|
const THROTTLE_MS = 100 // Update UI max once per 100ms
|
|
|
|
await brain.import(buffer, {
|
|
onProgress: (progress) => {
|
|
const now = Date.now()
|
|
if (now - lastUIUpdate < THROTTLE_MS && progress.stage !== 'complete') {
|
|
return // Skip this update
|
|
}
|
|
|
|
lastUIUpdate = now
|
|
updateUI(progress) // Only update every 100ms
|
|
}
|
|
})
|
|
```
|
|
|
|
**Note:** Brainy already throttles progress callbacks internally, but additional UI throttling can help with heavy rendering.
|
|
|
|
---
|
|
|
|
## 📝 Summary
|
|
|
|
✅ **All 7 formats** have consistent progress reporting
|
|
✅ **Real-time updates** during long imports (no more "0%" hangs)
|
|
✅ **Contextual messages** show exactly what's happening
|
|
✅ **Build reliable tools** with standardized progress callbacks
|
|
✅ **Problem SOLVED** - users see progress throughout import
|
|
|
|
**Files Modified:**
|
|
- 3 handlers: `csvHandler.ts`, `pdfHandler.ts`, `excelHandler.ts`
|
|
- 7 importers: `SmartCSVImporter.ts`, `SmartPDFImporter.ts`, `SmartExcelImporter.ts`, `SmartJSONImporter.ts`, `SmartMarkdownImporter.ts`, `SmartYAMLImporter.ts`, `SmartDOCXImporter.ts`
|
|
|
|
**Result:** Comprehensive, consistent progress tracking across ALL import formats!
|