feat: comprehensive import progress tracking for all 7 formats
Add real-time progress reporting throughout the entire import pipeline
with a standardized API that works across all supported formats.
Workshop Team Feature Request:
- Eliminates "0% complete" hangs during AI extraction
- Shows continuous progress with entities/sec, throughput, ETA
- Reports contextual messages ("Processing page 5 of 23")
- Standardized progress API for CSV, PDF, Excel, JSON, Markdown, YAML, DOCX
Core Changes:
- Add FormatHandlerProgressHooks interface for extensible progress
- Wire up all 3 binary format handlers (CSV, PDF, Excel) with 7+ progress points
- Wire up all 4 text format importers (JSON, Markdown, YAML, DOCX)
- Add ImportProgress interface with stage, message, counts, throughput, ETA
- ImportCoordinator normalizes all format progress to standard interface
CLI Improvements:
- Import command now uses brain.import() directly with full progress
- Add --include-vfs flag to find command (v4.4.0 compatibility)
- Add --confidence and --weight options to add command
Documentation:
- docs/guides/standard-import-progress.md - Universal API guide
- docs/guides/import-progress-implementation.md - Developer guide
- docs/guides/import-progress-examples.md - Practical examples
- JSDoc on brain.import() with universal handler examples
Result: ONE progress handler works for ALL 7 formats with zero format-specific code!
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
e7ea9c4e4b
commit
d5576ffb56
20 changed files with 3967 additions and 52 deletions
|
|
@ -2039,9 +2039,27 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
* groupBy: 'type', // Organize by entity type
|
||||
* preserveSource: true, // Keep original file
|
||||
*
|
||||
* // Progress tracking
|
||||
* onProgress: (p) => console.log(p.message)
|
||||
* // Progress tracking (v4.5.0 - STANDARDIZED FOR ALL 7 FORMATS!)
|
||||
* onProgress: (p) => {
|
||||
* console.log(`[${p.stage}] ${p.message}`)
|
||||
* console.log(`Entities: ${p.entities || 0}, Rels: ${p.relationships || 0}`)
|
||||
* if (p.throughput) console.log(`Rate: ${p.throughput.toFixed(1)}/sec`)
|
||||
* }
|
||||
* })
|
||||
* // THIS SAME HANDLER WORKS FOR CSV, PDF, Excel, JSON, Markdown, YAML, DOCX!
|
||||
* ```
|
||||
*
|
||||
* @example Universal Progress Handler (v4.5.0)
|
||||
* ```typescript
|
||||
* // ONE handler for ALL 7 formats - no format-specific code needed!
|
||||
* const universalProgress = (p) => {
|
||||
* updateUI(p.stage, p.message, p.entities, p.relationships)
|
||||
* }
|
||||
*
|
||||
* await brain.import(csvBuffer, { onProgress: universalProgress })
|
||||
* await brain.import(pdfBuffer, { onProgress: universalProgress })
|
||||
* await brain.import(excelBuffer, { onProgress: universalProgress })
|
||||
* // Works for JSON, Markdown, YAML, DOCX too!
|
||||
* ```
|
||||
*
|
||||
* @example Performance Tuning (Large Files)
|
||||
|
|
@ -2066,6 +2084,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
*
|
||||
* @see {@link https://brainy.dev/docs/api/import API Documentation}
|
||||
* @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
|
||||
* @see {@link https://brainy.dev/docs/guides/standard-import-progress Standard Progress API (v4.5.0)}
|
||||
*
|
||||
* @remarks
|
||||
* **⚠️ Breaking Changes from v3.x:**
|
||||
|
|
@ -2098,7 +2117,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
async import(
|
||||
source: Buffer | string | object,
|
||||
options?: {
|
||||
format?: 'excel' | 'pdf' | 'csv' | 'json' | 'markdown'
|
||||
format?: 'excel' | 'pdf' | 'csv' | 'json' | 'markdown' | 'yaml' | 'docx'
|
||||
vfsPath?: string
|
||||
groupBy?: 'type' | 'sheet' | 'flat' | 'custom'
|
||||
customGrouping?: (entity: any) => string
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue