feat: comprehensive import progress tracking for all 7 formats

Add real-time progress reporting throughout the entire import pipeline
with a standardized API that works across all supported formats.

Workshop Team Feature Request:
- Eliminates "0% complete" hangs during AI extraction
- Shows continuous progress with entities/sec, throughput, ETA
- Reports contextual messages ("Processing page 5 of 23")
- Standardized progress API for CSV, PDF, Excel, JSON, Markdown, YAML, DOCX

Core Changes:
- Add FormatHandlerProgressHooks interface for extensible progress
- Wire up all 3 binary format handlers (CSV, PDF, Excel) with 7+ progress points
- Wire up all 4 text format importers (JSON, Markdown, YAML, DOCX)
- Add ImportProgress interface with stage, message, counts, throughput, ETA
- ImportCoordinator normalizes all format progress to standard interface

CLI Improvements:
- Import command now uses brain.import() directly with full progress
- Add --include-vfs flag to find command (v4.4.0 compatibility)
- Add --confidence and --weight options to add command

Documentation:
- docs/guides/standard-import-progress.md - Universal API guide
- docs/guides/import-progress-implementation.md - Developer guide
- docs/guides/import-progress-examples.md - Practical examples
- JSDoc on brain.import() with universal handler examples

Result: ONE progress handler works for ALL 7 formats with zero format-specific code!

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
David Snelling 2025-10-24 14:45:46 -07:00
parent e7ea9c4e4b
commit d5576ffb56
20 changed files with 3967 additions and 52 deletions

View file

@ -174,9 +174,25 @@ export class SmartMarkdownImporter {
...options
}
// v4.5.0: Report parsing start
opts.onProgress({
processed: 0,
total: 0,
entities: 0,
relationships: 0
})
// Parse markdown into sections
const parsedSections = this.parseMarkdown(markdown, opts)
// v4.5.0: Report parsing complete
opts.onProgress({
processed: 0,
total: parsedSections.length,
entities: 0,
relationships: 0
})
// Process each section
const sections: MarkdownSection[] = []
const entityMap = new Map<string, string>()
@ -202,10 +218,20 @@ export class SmartMarkdownImporter {
})
}
// v4.5.0: Report completion
const totalEntities = sections.reduce((sum, s) => sum + s.entities.length, 0)
const totalRelationships = sections.reduce((sum, s) => sum + s.relationships.length, 0)
opts.onProgress({
processed: sections.length,
total: sections.length,
entities: totalEntities,
relationships: totalRelationships
})
return {
sectionsProcessed: sections.length,
entitiesExtracted: sections.reduce((sum, s) => sum + s.entities.length, 0),
relationshipsInferred: sections.reduce((sum, s) => sum + s.relationships.length, 0),
entitiesExtracted: totalEntities,
relationshipsInferred: totalRelationships,
sections,
entityMap,
processingTime: Date.now() - startTime,